2021-03-09T09:29:30.105 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-09T09:29:30.109 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_kchai@teuthology --name kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi --description rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/pool-create-delete} --verbose --archive /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949128 -- /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949128/orig.config.yaml 2021-03-09T09:29:30.165 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-09T09:29:30.231 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949128 branch: wip-kefu-testing-2021-03-09-1428 description: rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/pool-create-delete} email: tchaikov@gmail.com first_in_suite: false job_id: '5949128' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-kefu-testing-2021-03-09-1428 ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - slow request - overall HEALTH_ - \(POOL_APP_NOT_ENABLED\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) mon_bind_msgr2: false sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2021-03-09-1428 sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-03-09-1428 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 targets: smithi143.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYlkzOm2rvOuAmgVE+jCvsGyn0dB2+sZy0kvo8mvAZPjsSjFrbSrGBBVn3OOlfKALGntG/EG+3SGHpwX2fHqnKIaYTGNkRg6lS9f334YeZj+Md4Oa1YJt6H9wtQQr74hQEUAkIkRRBepq5aclptcLucERYMf/81Fg2m5hnx22QyS5/rjLiz/AMSaTTKSdgSYQLwKIlarQo1F/MEpFRHE4vmQAXbXGQn/iwKaiuwk6J0ojL1uW2dZ9P7tsxawZOJzQGXjzwClAsMklbV/3U4ekSPT/lDuG4bB7YxFzKulUYLlGmcvQOa5TGaGbg2mqNmrBg/mdx2Ho5ggc/9VfirBM7 smithi173.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOyMvWSMPEqNIM7iCReHZ79iR8+a8F4KT2ENGt0CXrU73TGJZOjKHfNG5iIUACUllwOLUrQ8+213akipy58vlWXu1JsR69kPvSGmXnl4KbeUlEw4SwYw4DuFN7m3P3eD0UtW304pB0pkKTDcp2XJ1Ew/BXjaxUViOn6/dICx8/UrgNhv9qDYg8Ulg5cLf6swndZY9uFrDxlyeCpodHO2m24fIeRAvdABWriVkEEsmoPG8ieMeVEOoLBV3vk5Y6GQ6xGbUkX/BMKcURnaMx37GMiEnpuXtCizOrcRw3nw298fx1sryeOybHu6BxYccx9DiHZWuWJbMss2dcaIFac7Wd tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology_branch: master teuthology_sha1: 3082387bba74fcd24c9700593d10418152d53c97 timestamp: 2021-03-09_08:22:40 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.25973 2021-03-09T09:29:30.232 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa; will attempt to use it 2021-03-09T09:29:30.245 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa/tasks 2021-03-09T09:29:30.245 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-09T09:29:30.248 INFO:teuthology.task.internal:Checking packages... 2021-03-09T09:29:30.333 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '89fb622b10ca2e58fe0c913a8956377a886e2ab4' 2021-03-09T09:29:30.334 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-09T09:29:30.334 INFO:teuthology.packaging:ref: None 2021-03-09T09:29:30.335 INFO:teuthology.packaging:tag: None 2021-03-09T09:29:30.335 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-03-09-1428 2021-03-09T09:29:30.335 INFO:teuthology.packaging:sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:29:30.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-kefu-testing-2021-03-09-1428 2021-03-09T09:29:30.495 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:29:30.495 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-09T09:29:30.506 INFO:teuthology.task.internal:no buildpackages task found 2021-03-09T09:29:30.506 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-09T09:29:30.532 INFO:teuthology.task.internal:Saving configuration 2021-03-09T09:29:30.550 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-09T09:29:30.582 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-09T09:29:30.601 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 09:23:55.224469', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949128', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJuBbZe+PWQcGHdz0Ha5RzARIyE6wvB71yfKt6z5M/yu', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi143.front.sepia.ceph.com'} 2021-03-09T09:29:30.617 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 09:23:55.224437', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949128', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN111No9LnG11BlrPGFjRGLueQAQFcKkUyWpSsRIyUsA', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi173.front.sepia.ceph.com'} 2021-03-09T09:29:30.617 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-09T09:29:30.625 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2021-03-09T09:29:30.625 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2021-03-09T09:29:30.625 INFO:teuthology.run_tasks:Running task console_log... 2021-03-09T09:29:30.696 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7fab2c89d8>, signals=[15]) 2021-03-09T09:29:30.700 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-09T09:29:30.706 INFO:teuthology.task.internal:Opening connections... 2021-03-09T09:29:30.707 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2021-03-09T09:29:30.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T09:29:30.785 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2021-03-09T09:29:30.786 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T09:29:30.859 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-09T09:29:30.866 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2021-03-09T09:29:30.977 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2021-03-09T09:29:30.978 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2021-03-09T09:29:31.024 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="Ubuntu" 2021-03-09T09:29:31.024 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-09T09:29:31.025 INFO:teuthology.orchestra.run.smithi143.stdout:ID=ubuntu 2021-03-09T09:29:31.025 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE=debian 2021-03-09T09:29:31.025 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-09T09:29:31.025 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="18.04" 2021-03-09T09:29:31.026 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-09T09:29:31.026 INFO:teuthology.orchestra.run.smithi143.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-09T09:29:31.026 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-09T09:29:31.027 INFO:teuthology.orchestra.run.smithi143.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-09T09:29:31.027 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_CODENAME=bionic 2021-03-09T09:29:31.027 INFO:teuthology.orchestra.run.smithi143.stdout:UBUNTU_CODENAME=bionic 2021-03-09T09:29:31.028 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2021-03-09T09:29:31.097 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2021-03-09T09:29:31.106 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2021-03-09T09:29:31.106 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2021-03-09T09:29:31.154 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="Ubuntu" 2021-03-09T09:29:31.155 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-09T09:29:31.155 INFO:teuthology.orchestra.run.smithi173.stdout:ID=ubuntu 2021-03-09T09:29:31.155 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE=debian 2021-03-09T09:29:31.155 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-09T09:29:31.156 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="18.04" 2021-03-09T09:29:31.156 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-09T09:29:31.156 INFO:teuthology.orchestra.run.smithi173.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-09T09:29:31.156 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-09T09:29:31.157 INFO:teuthology.orchestra.run.smithi173.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-09T09:29:31.157 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_CODENAME=bionic 2021-03-09T09:29:31.157 INFO:teuthology.orchestra.run.smithi173.stdout:UBUNTU_CODENAME=bionic 2021-03-09T09:29:31.158 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2021-03-09T09:29:31.173 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-09T09:29:31.183 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-09T09:29:31.207 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-09T09:29:31.207 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2021-03-09T09:29:31.211 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2021-03-09T09:29:31.216 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-09T09:29:31.224 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-09T09:29:31.224 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2021-03-09T09:29:31.260 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2021-03-09T09:29:31.269 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-09T09:29:31.315 INFO:teuthology.run_tasks:Running task kernel... 2021-03-09T09:29:31.332 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mon.d': {'kdb': True, 'sha1': 'distro'}, 'mon.e': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'mon.f': {'kdb': True, 'sha1': 'distro'}, 'mon.g': {'kdb': True, 'sha1': 'distro'}, 'mon.h': {'kdb': True, 'sha1': 'distro'}, 'mon.i': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-03-09T09:29:31.333 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.f': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-09T09:29:31.333 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2021-03-09T09:29:31.339 INFO:teuthology.orchestra.run.smithi143.stdout:4.15.0-136-generic 2021-03-09T09:29:31.340 INFO:teuthology.task.kernel:Running kernel on smithi143: 4.15.0-136-generic 2021-03-09T09:29:31.340 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get clean 2021-03-09T09:29:31.442 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2021-03-09T09:29:31.725 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T09:29:31.726 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-09T09:29:31.778 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-09T09:29:32.066 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-09T09:29:32.124 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,573 kB] 2021-03-09T09:29:32.255 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,228 kB] 2021-03-09T09:29:32.555 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [926 kB] 2021-03-09T09:29:32.575 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [302 kB] 2021-03-09T09:29:32.579 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-09T09:29:32.580 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [245 kB] 2021-03-09T09:29:32.586 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [31.7 kB] 2021-03-09T09:29:32.587 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,113 kB] 2021-03-09T09:29:32.600 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,913 kB] 2021-03-09T09:29:32.654 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-09T09:29:32.674 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [249 kB] 2021-03-09T09:29:32.677 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-09T09:29:32.677 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-09T09:29:32.678 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-09T09:29:32.703 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB] 2021-03-09T09:29:32.714 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-09T09:29:32.715 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [35.3 kB] 2021-03-09T09:29:32.716 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-09T09:29:32.767 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-09T09:29:32.802 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-09T09:29:32.803 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-09T09:29:32.804 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-09T09:29:34.282 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 12.5 MB in 2s (5,111 kB/s) 2021-03-09T09:29:35.738 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2021-03-09T09:29:35.785 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-09T09:29:35.863 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2021-03-09T09:29:36.120 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2021-03-09T09:29:36.121 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2021-03-09T09:29:36.200 INFO:teuthology.orchestra.run.smithi143.stderr:E: Unable to locate package linux-image-current-generic 2021-03-09T09:29:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-09T09:29:36.202 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-09T09:29:36.281 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2021-03-09T09:29:36.511 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2021-03-09T09:29:36.512 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2021-03-09T09:29:36.716 INFO:teuthology.orchestra.run.smithi143.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-09T09:29:36.716 INFO:teuthology.orchestra.run.smithi143.stdout:linux-image-generic set to manually installed. 2021-03-09T09:29:36.717 INFO:teuthology.orchestra.run.smithi143.stdout:0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T09:29:36.718 DEBUG:teuthology.orchestra.run.smithi143:> dpkg -s linux-image-generic 2021-03-09T09:29:36.739 INFO:teuthology.orchestra.run.smithi143.stdout:Package: linux-image-generic 2021-03-09T09:29:36.739 INFO:teuthology.orchestra.run.smithi143.stdout:Status: install ok installed 2021-03-09T09:29:36.739 INFO:teuthology.orchestra.run.smithi143.stdout:Priority: optional 2021-03-09T09:29:36.739 INFO:teuthology.orchestra.run.smithi143.stdout:Section: kernel 2021-03-09T09:29:36.740 INFO:teuthology.orchestra.run.smithi143.stdout:Installed-Size: 17 2021-03-09T09:29:36.740 INFO:teuthology.orchestra.run.smithi143.stdout:Maintainer: Ubuntu Kernel Team 2021-03-09T09:29:36.740 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: amd64 2021-03-09T09:29:36.740 INFO:teuthology.orchestra.run.smithi143.stdout:Source: linux-meta 2021-03-09T09:29:36.740 INFO:teuthology.orchestra.run.smithi143.stdout:Version: 4.15.0.136.123 2021-03-09T09:29:36.741 INFO:teuthology.orchestra.run.smithi143.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-09T09:29:36.741 INFO:teuthology.orchestra.run.smithi143.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-09T09:29:36.741 INFO:teuthology.orchestra.run.smithi143.stdout:Recommends: thermald 2021-03-09T09:29:36.741 INFO:teuthology.orchestra.run.smithi143.stdout:Description: Generic Linux kernel image 2021-03-09T09:29:36.742 INFO:teuthology.orchestra.run.smithi143.stdout: This package will always depend on the latest generic kernel image 2021-03-09T09:29:36.742 INFO:teuthology.orchestra.run.smithi143.stdout: available. 2021-03-09T09:29:36.742 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-09T09:29:36.743 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2021-03-09T09:29:36.749 INFO:teuthology.orchestra.run.smithi173.stdout:4.15.0-136-generic 2021-03-09T09:29:36.749 INFO:teuthology.task.kernel:Running kernel on smithi173: 4.15.0-136-generic 2021-03-09T09:29:36.749 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get clean 2021-03-09T09:29:36.869 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get update 2021-03-09T09:29:37.142 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T09:29:37.143 INFO:teuthology.orchestra.run.smithi173.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-09T09:29:37.199 INFO:teuthology.orchestra.run.smithi173.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-09T09:29:37.474 INFO:teuthology.orchestra.run.smithi173.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-09T09:29:37.521 INFO:teuthology.orchestra.run.smithi173.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,573 kB] 2021-03-09T09:29:37.651 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,913 kB] 2021-03-09T09:29:37.947 INFO:teuthology.orchestra.run.smithi173.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [926 kB] 2021-03-09T09:29:37.966 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [302 kB] 2021-03-09T09:29:37.972 INFO:teuthology.orchestra.run.smithi173.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [245 kB] 2021-03-09T09:29:37.976 INFO:teuthology.orchestra.run.smithi173.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-09T09:29:37.977 INFO:teuthology.orchestra.run.smithi173.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [31.7 kB] 2021-03-09T09:29:37.978 INFO:teuthology.orchestra.run.smithi173.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-09T09:29:38.045 INFO:teuthology.orchestra.run.smithi173.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,113 kB] 2021-03-09T09:29:38.061 INFO:teuthology.orchestra.run.smithi173.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,228 kB] 2021-03-09T09:29:38.072 INFO:teuthology.orchestra.run.smithi173.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [249 kB] 2021-03-09T09:29:38.078 INFO:teuthology.orchestra.run.smithi173.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-09T09:29:38.078 INFO:teuthology.orchestra.run.smithi173.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-09T09:29:38.079 INFO:teuthology.orchestra.run.smithi173.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-09T09:29:38.099 INFO:teuthology.orchestra.run.smithi173.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB] 2021-03-09T09:29:38.107 INFO:teuthology.orchestra.run.smithi173.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-09T09:29:38.107 INFO:teuthology.orchestra.run.smithi173.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [35.3 kB] 2021-03-09T09:29:38.108 INFO:teuthology.orchestra.run.smithi173.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-09T09:29:38.171 INFO:teuthology.orchestra.run.smithi173.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-09T09:29:38.216 INFO:teuthology.orchestra.run.smithi173.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-09T09:29:38.217 INFO:teuthology.orchestra.run.smithi173.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-09T09:29:38.217 INFO:teuthology.orchestra.run.smithi173.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-09T09:29:39.536 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 12.5 MB in 2s (5,097 kB/s) 2021-03-09T09:29:41.001 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2021-03-09T09:29:41.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-09T09:29:41.128 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2021-03-09T09:29:41.359 INFO:teuthology.orchestra.run.smithi173.stdout:Building dependency tree... 2021-03-09T09:29:41.360 INFO:teuthology.orchestra.run.smithi173.stdout:Reading state information... 2021-03-09T09:29:41.439 INFO:teuthology.orchestra.run.smithi173.stderr:E: Unable to locate package linux-image-current-generic 2021-03-09T09:29:41.440 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-09T09:29:41.441 DEBUG:teuthology.orchestra.run.smithi173:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-09T09:29:41.522 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2021-03-09T09:29:41.780 INFO:teuthology.orchestra.run.smithi173.stdout:Building dependency tree... 2021-03-09T09:29:41.780 INFO:teuthology.orchestra.run.smithi173.stdout:Reading state information... 2021-03-09T09:29:41.985 INFO:teuthology.orchestra.run.smithi173.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-09T09:29:41.986 INFO:teuthology.orchestra.run.smithi173.stdout:linux-image-generic set to manually installed. 2021-03-09T09:29:41.986 INFO:teuthology.orchestra.run.smithi173.stdout:0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T09:29:41.987 DEBUG:teuthology.orchestra.run.smithi173:> dpkg -s linux-image-generic 2021-03-09T09:29:42.002 INFO:teuthology.orchestra.run.smithi173.stdout:Package: linux-image-generic 2021-03-09T09:29:42.002 INFO:teuthology.orchestra.run.smithi173.stdout:Status: install ok installed 2021-03-09T09:29:42.002 INFO:teuthology.orchestra.run.smithi173.stdout:Priority: optional 2021-03-09T09:29:42.003 INFO:teuthology.orchestra.run.smithi173.stdout:Section: kernel 2021-03-09T09:29:42.003 INFO:teuthology.orchestra.run.smithi173.stdout:Installed-Size: 17 2021-03-09T09:29:42.003 INFO:teuthology.orchestra.run.smithi173.stdout:Maintainer: Ubuntu Kernel Team 2021-03-09T09:29:42.003 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: amd64 2021-03-09T09:29:42.004 INFO:teuthology.orchestra.run.smithi173.stdout:Source: linux-meta 2021-03-09T09:29:42.004 INFO:teuthology.orchestra.run.smithi173.stdout:Version: 4.15.0.136.123 2021-03-09T09:29:42.004 INFO:teuthology.orchestra.run.smithi173.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-09T09:29:42.004 INFO:teuthology.orchestra.run.smithi173.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-09T09:29:42.005 INFO:teuthology.orchestra.run.smithi173.stdout:Recommends: thermald 2021-03-09T09:29:42.005 INFO:teuthology.orchestra.run.smithi173.stdout:Description: Generic Linux kernel image 2021-03-09T09:29:42.005 INFO:teuthology.orchestra.run.smithi173.stdout: This package will always depend on the latest generic kernel image 2021-03-09T09:29:42.005 INFO:teuthology.orchestra.run.smithi173.stdout: available. 2021-03-09T09:29:42.006 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-09T09:29:42.006 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-03-09T09:29:42.006 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T09:29:42.118 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2021-03-09T09:29:42.152 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2021-03-09T09:29:42.153 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T09:29:42.168 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2021-03-09T09:29:42.190 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-09T09:29:42.197 INFO:teuthology.task.internal:Creating test directory... 2021-03-09T09:29:42.199 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T09:29:42.201 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T09:29:42.238 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-09T09:29:42.248 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-09T09:29:42.270 INFO:teuthology.task.internal:Creating archive directory... 2021-03-09T09:29:42.271 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T09:29:42.273 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T09:29:42.321 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-09T09:29:42.331 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-09T09:29:42.331 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T09:29:42.339 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T09:29:42.362 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T09:29:42.370 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T09:29:42.376 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T09:29:42.386 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T09:29:42.387 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-09T09:29:42.397 INFO:teuthology.task.internal:Configuring sudo... 2021-03-09T09:29:42.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T09:29:42.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T09:29:42.443 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-09T09:29:42.460 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-09T09:29:42.461 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T09:29:42.471 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T09:29:42.490 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T09:29:42.524 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T09:29:42.572 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:29:42.573 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T09:29:42.631 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T09:29:42.638 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T09:29:42.686 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:29:42.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T09:29:42.747 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2021-03-09T09:29:42.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2021-03-09T09:29:42.830 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-09T09:29:42.838 INFO:teuthology.task.internal:Starting timer... 2021-03-09T09:29:42.838 INFO:teuthology.run_tasks:Running task pcp... 2021-03-09T09:29:42.863 INFO:teuthology.run_tasks:Running task selinux... 2021-03-09T09:29:42.878 DEBUG:teuthology.task.selinux:Excluding smithi143: OS 'ubuntu' does not support SELinux 2021-03-09T09:29:42.879 DEBUG:teuthology.task.selinux:Excluding smithi173: OS 'ubuntu' does not support SELinux 2021-03-09T09:29:42.879 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-09T09:29:42.879 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-03-09T09:29:42.880 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-09T09:29:42.880 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-09T09:29:42.898 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-03-09T09:29:42.899 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-09T09:29:43.229 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-09T09:29:43.237 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi143.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-09T09:29:44.138 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-09T09:29:45.872 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-09T09:29:48.325 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-09T09:29:48.382 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:48 +0000 (0:00:00.083) 0:00:00.083 ********* 2021-03-09T09:29:48.400 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:48 +0000 (0:00:00.017) 0:00:00.101 ********* 2021-03-09T09:29:50.134 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-09T09:29:50.136 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi173.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi173.front.sepia.ceph.com] 2021-03-09T09:29:50.193 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:50 +0000 (0:00:01.792) 0:00:01.894 ********* 2021-03-09T09:29:50.230 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi143.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi143.front.sepia.ceph.com] 2021-03-09T09:29:50.282 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:50 +0000 (0:00:00.089) 0:00:01.984 ********* 2021-03-09T09:29:50.466 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-09T09:29:50.468 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T09:29:50.491 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:50 +0000 (0:00:00.209) 0:00:02.193 ********* 2021-03-09T09:29:50.512 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:50 +0000 (0:00:00.020) 0:00:02.214 ********* 2021-03-09T09:29:51.179 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-03-09T09:29:51.181 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-09T09:29:51.207 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:51 +0000 (0:00:00.694) 0:00:02.908 ********* 2021-03-09T09:29:51.215 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-09T09:29:51.248 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:51 +0000 (0:00:00.040) 0:00:02.949 ********* 2021-03-09T09:29:51.586 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-03-09T09:29:51.588 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.066450", "end": "2021-03-09 09:29:51.666691", "rc": 0, "start": "2021-03-09 09:29:51.600241", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.070257", "end": "2021-03-09 09:29:51.697101", "rc": 0, "start": "2021-03-09 09:29:51.626844", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-09T09:29:51.610 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:51 +0000 (0:00:00.362) 0:00:03.311 ********* 2021-03-09T09:29:51.639 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:51 +0000 (0:00:00.029) 0:00:03.340 ********* 2021-03-09T09:29:51.866 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-03-09T09:29:51.868 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:29:51.894 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:51 +0000 (0:00:00.255) 0:00:03.595 ********* 2021-03-09T09:29:51.901 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:29:51.926 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:51 +0000 (0:00:00.031) 0:00:03.627 ********* 2021-03-09T09:29:52.743 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-03-09T09:29:52.745 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282191.9609146-30725-242471730902175/source", "state": "file", "uid": 0} changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282191.9320798-30722-86211752284232/source", "state": "file", "uid": 0} 2021-03-09T09:29:52.767 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:52 +0000 (0:00:00.841) 0:00:04.469 ********* 2021-03-09T09:29:52.788 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:52 +0000 (0:00:00.020) 0:00:04.489 ********* 2021-03-09T09:29:53.234 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-03-09T09:29:53.235 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-09T09:29:53.237 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T09:29:53.263 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-09T09:29:53.264 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T09:29:53.342 INFO:teuthology.task.ansible.out: 2021-03-09T09:29:53.344 INFO:teuthology.task.ansible.out:PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-09T09:29:53.347 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-09T09:29:53.510 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:53 +0000 (0:00:00.722) 0:00:05.211 ********* 2021-03-09T09:29:53.533 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:53 +0000 (0:00:00.022) 0:00:05.234 ********* 2021-03-09T09:29:54.294 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi143.front.sepia.ceph.com] 2021-03-09T09:29:54.304 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] 2021-03-09T09:29:54.391 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-03-09T09:29:54.393 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:54 +0000 (0:00:00.858) 0:00:06.093 ********* 2021-03-09T09:29:54.416 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:54 +0000 (0:00:00.024) 0:00:06.117 ********* 2021-03-09T09:29:54.850 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-09T09:29:54.852 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.184803", "end": "2021-03-09 09:29:54.822788", "rc": 0, "start": "2021-03-09 09:29:54.637985", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.186815", "end": "2021-03-09 09:29:54.829574", "rc": 0, "start": "2021-03-09 09:29:54.642759", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-09T09:29:54.874 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:54 +0000 (0:00:00.458) 0:00:06.575 ********* 2021-03-09T09:29:54.896 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:54 +0000 (0:00:00.021) 0:00:06.597 ********* 2021-03-09T09:29:55.073 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-09T09:29:55.076 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013214", "end": "2021-03-09 09:29:55.179806", "failed_when_result": false, "rc": 0, "start": "2021-03-09 09:29:55.166592", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T09:29:55.102 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.205) 0:00:06.803 ********* 2021-03-09T09:29:55.171 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-09T09:29:55.173 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.195 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.093) 0:00:06.896 ********* 2021-03-09T09:29:55.264 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-03-09T09:29:55.266 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.289 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.093) 0:00:06.990 ********* 2021-03-09T09:29:55.316 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-09T09:29:55.318 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.311219", "end": "2021-03-09 09:29:55.400078", "failed_when_result": false, "rc": 0, "start": "2021-03-09 09:29:55.088859", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T09:29:55.343 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.053) 0:00:07.044 ********* 2021-03-09T09:29:55.362 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-09T09:29:55.364 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.388 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.044) 0:00:07.089 ********* 2021-03-09T09:29:55.415 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-09T09:29:55.417 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.446 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.058) 0:00:07.147 ********* 2021-03-09T09:29:55.466 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-09T09:29:55.467 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.494 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.047) 0:00:07.195 ********* 2021-03-09T09:29:55.524 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-03-09T09:29:55.525 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.549 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.054) 0:00:07.250 ********* 2021-03-09T09:29:55.567 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-09T09:29:55.569 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.592 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.042) 0:00:07.293 ********* 2021-03-09T09:29:55.619 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.643 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.051) 0:00:07.345 ********* 2021-03-09T09:29:55.722 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-09T09:29:55.724 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.745 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.101) 0:00:07.446 ********* 2021-03-09T09:29:55.782 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-03-09T09:29:55.784 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:29:55.807 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.061) 0:00:07.508 ********* 2021-03-09T09:29:55.836 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-09T09:29:55.838 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:29:55.865 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:55 +0000 (0:00:00.057) 0:00:07.566 ********* 2021-03-09T09:29:56.059 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T09:29:56.061 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi143.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:29:56.087 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:29:56 +0000 (0:00:00.222) 0:00:07.788 ********* 2021-03-09T09:29:56.208 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T09:29:56.209 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T09:29:56.334 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-09T09:29:56.346 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T09:29:56.471 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T09:29:56.482 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T09:29:56.620 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T09:29:56.631 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T09:29:56.767 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T09:29:56.778 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T09:29:56.903 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T09:29:56.915 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T09:29:57.050 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T09:29:57.062 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T09:29:57.187 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T09:29:57.189 INFO:teuthology.task.ansible.out: 2021-03-09T09:29:57.199 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T09:29:57.201 INFO:teuthology.task.ansible.out: 2021-03-09T09:29:57.336 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T09:29:57.337 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T09:29:57.472 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T09:29:57.484 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T09:29:57.620 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T09:29:57.621 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T09:29:57.757 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T09:29:57.758 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T09:29:57.894 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T09:29:57.905 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T09:29:58.041 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T09:29:58.042 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T09:29:58.167 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T09:29:58.179 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T09:29:58.314 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T09:29:58.326 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T09:29:58.452 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T09:29:58.463 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T09:29:58.600 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T09:29:58.611 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T09:29:58.748 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T09:29:58.749 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T09:29:58.885 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T09:29:58.886 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T09:29:59.032 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T09:29:59.034 INFO:teuthology.task.ansible.out: ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T09:29:59.169 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T09:29:59.170 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T09:29:59.317 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T09:29:59.318 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T09:29:59.463 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T09:29:59.464 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T09:29:59.600 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T09:29:59.601 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T09:29:59.747 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T09:29:59.748 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T09:29:59.884 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T09:29:59.885 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T09:30:00.021 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T09:30:00.033 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T09:30:00.168 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T09:30:00.169 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T09:30:00.316 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T09:30:00.317 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T09:30:00.454 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T09:30:00.455 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T09:30:00.591 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T09:30:00.603 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T09:30:00.739 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T09:30:00.750 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T09:30:00.887 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T09:30:00.890 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T09:30:01.025 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T09:30:01.036 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T09:30:01.168 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T09:30:01.180 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T09:30:01.318 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T09:30:01.326 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T09:30:01.464 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T09:30:01.465 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T09:30:01.602 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T09:30:01.605 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T09:30:01.752 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T09:30:01.754 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T09:30:01.892 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T09:30:01.893 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T09:30:02.031 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T09:30:02.033 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T09:30:02.181 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T09:30:02.182 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T09:30:02.329 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T09:30:02.330 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T09:30:02.478 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T09:30:02.479 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T09:30:02.626 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T09:30:02.628 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T09:30:02.774 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T09:30:02.776 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T09:30:02.912 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T09:30:02.923 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T09:30:03.060 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T09:30:03.061 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T09:30:03.208 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T09:30:03.210 INFO:teuthology.task.ansible.out: ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T09:30:03.211 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:03.349 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T09:30:03.351 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T09:30:03.500 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T09:30:03.501 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T09:30:03.639 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T09:30:03.641 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T09:30:03.642 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:03.782 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T09:30:03.784 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T09:30:03.920 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T09:30:03.931 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T09:30:04.073 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T09:30:04.074 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T09:30:04.211 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T09:30:04.222 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T09:30:04.348 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T09:30:04.360 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T09:30:04.362 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:04.497 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T09:30:04.520 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T09:30:04.656 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T09:30:04.669 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T09:30:04.806 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T09:30:04.828 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T09:30:04.954 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T09:30:05.083 INFO:teuthology.task.ansible.out: ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T09:30:05.128 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T09:30:05.139 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T09:30:05.266 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T09:30:05.289 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T09:30:05.291 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:05.423 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T09:30:05.436 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T09:30:05.573 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T09:30:05.585 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T09:30:05.721 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T09:30:05.722 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T09:30:05.868 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T09:30:05.869 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T09:30:06.015 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T09:30:06.017 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T09:30:06.152 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T09:30:06.163 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T09:30:06.299 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T09:30:06.301 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T09:30:06.447 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T09:30:06.448 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T09:30:06.585 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T09:30:06.586 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T09:30:06.721 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T09:30:06.733 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T09:30:06.870 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T09:30:06.871 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T09:30:07.006 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T09:30:07.007 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T09:30:07.154 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T09:30:07.156 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T09:30:07.290 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T09:30:07.302 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T09:30:07.439 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T09:30:07.450 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T09:30:07.585 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T09:30:07.597 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T09:30:07.733 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T09:30:07.734 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:07.736 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T09:30:07.871 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T09:30:07.873 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T09:30:08.020 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T09:30:08.031 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T09:30:08.168 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T09:30:08.179 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T09:30:08.304 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T09:30:08.326 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T09:30:08.443 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T09:30:08.465 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T09:30:08.591 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T09:30:08.613 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T09:30:08.738 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T09:30:08.760 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T09:30:08.886 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T09:30:08.897 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T09:30:09.033 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T09:30:09.045 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T09:30:09.181 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T09:30:09.192 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T09:30:09.330 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T09:30:09.331 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T09:30:09.481 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T09:30:09.483 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:09.505 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T09:30:09.662 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T09:30:09.696 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T09:30:09.833 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T09:30:09.886 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T09:30:09.981 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T09:30:10.035 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T09:30:10.130 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T09:30:10.205 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T09:30:10.279 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T09:30:10.384 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T09:30:10.438 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T09:30:10.553 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T09:30:10.587 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T09:30:10.715 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T09:30:10.748 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T09:30:10.875 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T09:30:10.897 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T09:30:11.034 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T09:30:11.045 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T09:30:11.184 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T09:30:11.196 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T09:30:11.332 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T09:30:11.344 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T09:30:11.481 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T09:30:11.482 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T09:30:11.618 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T09:30:11.629 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T09:30:11.777 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T09:30:11.779 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T09:30:11.915 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T09:30:11.926 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T09:30:12.063 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T09:30:12.064 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T09:30:12.201 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T09:30:12.213 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T09:30:12.214 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:12.349 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T09:30:12.350 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T09:30:12.485 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T09:30:12.486 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T09:30:12.633 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T09:30:12.634 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T09:30:12.770 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T09:30:12.781 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T09:30:12.918 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T09:30:12.920 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T09:30:13.066 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T09:30:13.067 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T09:30:13.215 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T09:30:13.216 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T09:30:13.352 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T09:30:13.353 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T09:30:13.499 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T09:30:13.511 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T09:30:13.647 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T09:30:13.648 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T09:30:13.794 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T09:30:13.795 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T09:30:13.932 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T09:30:13.933 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T09:30:14.080 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T09:30:14.081 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T09:30:14.217 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T09:30:14.229 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T09:30:14.231 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:14.355 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T09:30:14.377 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T09:30:14.504 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T09:30:14.527 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T09:30:14.643 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T09:30:14.665 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T09:30:14.791 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T09:30:14.813 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T09:30:14.929 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T09:30:14.962 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T09:30:15.099 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T09:30:15.214 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T09:30:15.247 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T09:30:15.362 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T09:30:15.384 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T09:30:15.511 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T09:30:15.533 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T09:30:15.535 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:15.649 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T09:30:15.650 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:15.682 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T09:30:15.798 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T09:30:15.820 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T09:30:15.935 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T09:30:15.967 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T09:30:16.083 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T09:30:16.107 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T09:30:16.224 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T09:30:16.371 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T09:30:16.403 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T09:30:16.519 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:16 +0000 (0:00:20.432) 0:00:28.220 ********* 2021-03-09T09:30:16.527 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T09:30:16.744 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-09T09:30:16.767 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:16 +0000 (0:00:00.248) 0:00:28.468 ********* 2021-03-09T09:30:16.790 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:16 +0000 (0:00:00.022) 0:00:28.491 ********* 2021-03-09T09:30:16.877 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:16 +0000 (0:00:00.086) 0:00:28.578 ********* 2021-03-09T09:30:16.886 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-09T09:30:16.889 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T09:30:16.890 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T09:30:16.892 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T09:30:16.894 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T09:30:16.895 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T09:30:16.897 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T09:30:16.899 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a 2021-03-09T09:30:16.899 INFO:teuthology.task.ansible.out:4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T09:30:16.901 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T09:30:16.902 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T09:30:16.904 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T09:30:16.905 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl 2021-03-09T09:30:16.906 INFO:teuthology.task.ansible.out:@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T09:30:16.907 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T09:30:16.909 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T09:30:16.939 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:16 +0000 (0:00:00.061) 0:00:28.640 ********* 2021-03-09T09:30:16.980 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T09:30:16.982 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T09:30:16.984 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T09:30:16.985 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T09:30:16.987 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T09:30:16.989 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T09:30:16.990 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T09:30:16.992 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T09:30:16.993 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T09:30:16.995 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T09:30:16.997 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T09:30:16.998 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T09:30:17.000 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174f 2021-03-09T09:30:17.000 INFO:teuthology.task.ansible.out:e16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T09:30:17.037 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:17 +0000 (0:00:00.097) 0:00:28.738 ********* 2021-03-09T09:30:17.812 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Clone the keys repo (5 retries left). 2021-03-09T09:30:17.823 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-09T09:30:17.825 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T09:30:17.848 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:17 +0000 (0:00:00.811) 0:00:29.549 ********* 2021-03-09T09:30:18.075 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:18.077 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T09:30:18.212 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T09:30:18.348 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T09:30:18.484 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-09T09:30:18.485 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-09T09:30:18.620 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T09:30:18.745 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-09T09:30:18.747 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-09T09:30:18.881 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T09:30:19.006 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-09T09:30:19.153 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-09T09:30:19.164 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:19.289 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T09:30:19.291 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-09T09:30:19.425 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T09:30:19.427 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-09T09:30:19.436 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:19.447 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:19.449 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:19.575 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-09T09:30:19.710 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T09:30:19.712 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T09:30:19.845 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T09:30:19.982 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-09T09:30:19.984 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-09T09:30:20.117 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-09T09:30:20.254 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T09:30:20.389 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T09:30:20.515 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-09T09:30:20.517 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-09T09:30:20.651 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T09:30:20.819 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T09:30:20.944 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T09:30:21.080 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T09:30:21.081 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T09:30:21.216 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T09:30:21.352 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T09:30:21.480 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T09:30:21.618 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T09:30:21.743 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T09:30:21.878 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T09:30:21.880 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-09T09:30:22.015 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T09:30:22.150 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T09:30:22.265 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T09:30:22.412 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T09:30:22.538 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T09:30:22.675 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T09:30:22.802 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T09:30:22.937 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T09:30:23.054 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T09:30:23.180 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T09:30:23.182 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T09:30:23.316 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T09:30:23.318 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T09:30:23.440 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T09:30:23.569 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T09:30:23.685 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T09:30:23.822 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T09:30:23.824 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T09:30:23.961 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T09:30:23.964 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T09:30:23.966 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T09:30:24.098 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T09:30:24.234 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T09:30:24.371 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T09:30:24.497 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T09:30:24.622 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T09:30:24.748 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T09:30:24.750 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-09T09:30:24.884 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T09:30:25.020 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T09:30:25.146 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T09:30:25.148 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-09T09:30:25.275 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T09:30:25.475 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T09:30:25.690 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T09:30:25.692 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-09T09:30:25.851 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-09T09:30:26.011 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T09:30:26.138 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T09:30:26.274 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T09:30:26.421 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T09:30:26.550 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-09T09:30:26.688 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-09T09:30:26.817 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-09T09:30:26.954 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-09T09:30:27.080 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-09T09:30:27.208 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T09:30:27.335 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T09:30:27.337 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:27.462 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T09:30:27.464 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:27.601 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T09:30:27.728 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T09:30:27.853 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T09:30:27.979 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T09:30:28.105 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T09:30:28.231 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T09:30:28.367 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T09:30:28.368 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi143.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 2, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T09:30:28.391 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:28 +0000 (0:00:10.543) 0:00:40.093 ********* 2021-03-09T09:30:28.493 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:28.494 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T09:30:28.496 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T09:30:28.621 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T09:30:28.643 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T09:30:28.749 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T09:30:28.792 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T09:30:28.886 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T09:30:28.928 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T09:30:29.012 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T09:30:29.065 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-09T09:30:29.067 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-09T09:30:29.140 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T09:30:29.142 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-09T09:30:29.204 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T09:30:29.268 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T09:30:29.269 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-09T09:30:29.343 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-09T09:30:29.345 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-09T09:30:29.397 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T09:30:29.482 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T09:30:29.514 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T09:30:29.620 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-09T09:30:29.655 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T09:30:29.770 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-09T09:30:29.772 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRX 2021-03-09T09:30:29.774 INFO:teuthology.task.ansible.out:dmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:29.909 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T09:30:29.910 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", 2021-03-09T09:30:29.912 INFO:teuthology.task.ansible.out: "user": "sseshasa", "validate_certs": true} 2021-03-09T09:30:30.046 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T09:30:30.047 INFO:teuthology.task.ansible.out:son", "validate_certs": true} ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T09:30:30.057 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:30.059 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:30.173 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T09:30:30.195 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-09T09:30:30.310 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T09:30:30.312 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-09T09:30:30.322 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T09:30:30.323 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T09:30:30.448 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T09:30:30.449 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T09:30:30.575 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T09:30:30.577 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-09T09:30:30.578 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-09T09:30:30.704 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-09T09:30:30.715 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-09T09:30:30.831 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T09:30:30.842 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T09:30:30.959 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T09:30:30.981 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T09:30:31.086 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T09:30:31.109 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-09T09:30:31.111 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-09T09:30:31.213 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T09:30:31.215 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T09:30:31.245 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T09:30:31.247 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-09T09:30:31.350 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T09:30:31.393 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T09:30:31.481 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T09:30:31.524 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T09:30:31.613 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T09:30:31.656 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T09:30:31.658 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T09:30:31.742 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T09:30:31.806 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T09:30:31.872 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-09T09:30:31.936 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T09:30:31.999 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T09:30:32.063 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T09:30:32.127 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T09:30:32.210 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T09:30:32.264 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T09:30:32.337 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T09:30:32.391 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T09:30:32.465 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T09:30:32.467 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-09T09:30:32.528 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T09:30:32.591 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T09:30:32.655 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T09:30:32.730 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T09:30:32.788 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T09:30:32.874 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T09:30:32.928 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T09:30:33.002 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T09:30:33.055 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T09:30:33.128 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T09:30:33.181 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T09:30:33.267 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T09:30:33.310 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T09:30:33.394 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T09:30:33.437 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T09:30:33.521 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T09:30:33.578 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T09:30:33.662 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T09:30:33.704 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T09:30:33.706 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T09:30:33.790 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T09:30:33.792 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T09:30:33.833 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T09:30:33.928 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T09:30:33.930 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T09:30:33.971 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T09:30:34.067 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T09:30:34.099 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T09:30:34.194 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T09:30:34.227 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T09:30:34.322 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T09:30:34.355 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T09:30:34.461 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T09:30:34.463 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T09:30:34.483 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-09T09:30:34.598 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T09:30:34.600 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T09:30:34.602 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T09:30:34.620 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T09:30:34.726 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T09:30:34.768 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T09:30:34.864 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T09:30:34.896 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T09:30:35.001 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T09:30:35.045 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T09:30:35.130 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T09:30:35.184 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T09:30:35.259 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T09:30:35.312 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T09:30:35.397 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T09:30:35.399 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-09T09:30:35.451 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T09:30:35.537 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T09:30:35.595 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T09:30:35.680 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T09:30:35.744 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T09:30:35.829 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T09:30:35.831 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-09T09:30:35.883 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T09:30:35.957 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T09:30:36.011 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options" 2021-03-09T09:30:36.013 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-09T09:30:36.085 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T09:30:36.149 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T09:30:36.231 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:36 +0000 (0:00:07.839) 0:00:47.932 ********* 2021-03-09T09:30:36.240 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T09:30:36.242 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-09T09:30:36.317 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:36.319 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.328 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.339 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.341 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.351 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:36.352 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450e 2021-03-09T09:30:36.354 INFO:teuthology.task.ansible.out:f0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.363 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.365 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.375 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.491 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:36.493 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T09:30:36.504 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:36.505 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-09T09:30:36.515 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.517 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.632 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:36.634 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T09:30:36.644 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:36.646 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-09T09:30:36.760 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:36.761 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T09:30:36.772 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:36.774 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-09T09:30:36.784 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.786 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.795 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.797 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.807 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.818 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.820 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.829 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.831 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.841 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.842 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.852 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.863 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.865 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.874 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.876 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.886 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.897 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.898 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:36.899 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T09:30:36.900 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.911 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.913 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.923 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.924 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.934 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.945 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.947 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.956 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.958 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.968 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.969 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.979 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.981 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:36.991 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.001 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.003 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.013 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.014 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.024 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:37.185 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepi 2021-03-09T09:30:37.187 INFO:teuthology.task.ansible.out:a.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 2021-03-09T09:30:37.189 INFO:teuthology.task.ansible.out: 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53 2021-03-09T09:30:37.191 INFO:teuthology.task.ansible.out:'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] 2021-03-09T09:30:37.193 INFO:teuthology.task.ansible.out: => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec4 2021-03-09T09:30:37.195 INFO:teuthology.task.ansible.out:8dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "r 2021-03-09T09:30:37.196 INFO:teuthology.task.ansible.out:ishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS 2021-03-09T09:30:37.198 INFO:teuthology.task.ansible.out:/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.206 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.218 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.219 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.231 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.235 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.242 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.253 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.254 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.264 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.265 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.276 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.287 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.289 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.299 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.311 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.312 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.322 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.324 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.334 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.345 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.347 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.356 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.357 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.368 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.379 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.381 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.391 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.402 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.403 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.413 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.425 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.426 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.437 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.439 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.449 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.460 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.461 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.471 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.472 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.474 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:37.484 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.486 INFO:teuthology.task.ansible.out: skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.496 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.507 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.509 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.518 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.520 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.530 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.541 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.543 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.552 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.554 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.564 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:37.566 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.577 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.588 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.589 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.599 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.601 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.611 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.622 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.664 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:37 +0000 (0:00:01.433) 0:00:49.365 ********* 2021-03-09T09:30:37.691 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:37.693 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T09:30:37.734 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-09T09:30:37.736 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:37.758 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:37 +0000 (0:00:00.094) 0:00:49.460 ********* 2021-03-09T09:30:37.830 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:37.832 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T09:30:37.968 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T09:30:37.980 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:37.981 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-09T09:30:38.107 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:38.109 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T09:30:38.118 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:38.120 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-09T09:30:38.244 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:38.246 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T09:30:38.266 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:38.268 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T09:30:38.382 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:38.384 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T09:30:38.404 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:38.405 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-09T09:30:38.520 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:38.522 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T09:30:38.552 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:38.554 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T09:30:38.647 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:38.649 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T09:30:38.690 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:38.692 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T09:30:38.786 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:38.788 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T09:30:38.829 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:38.831 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T09:30:38.914 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T09:30:38.967 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:38.969 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T09:30:39.052 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:39.053 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T09:30:39.055 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T09:30:39.096 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:39.097 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T09:30:39.183 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:39.184 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T09:30:39.238 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:39.240 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T09:30:39.314 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:39.316 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T09:30:39.388 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:39.390 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T09:30:39.452 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:39.454 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T09:30:39.537 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:39.539 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T09:30:39.591 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:39.592 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T09:30:39.671 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi173.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T09:30:39.724 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:39.726 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T09:30:39.728 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-09T09:30:39.832 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi173.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T09:30:39.875 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:39.877 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T09:30:39.879 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-09T09:30:39.982 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:39.984 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T09:30:40.026 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:40.028 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T09:30:40.122 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:40.124 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T09:30:40.155 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:40.157 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T09:30:40.273 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:40.276 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T09:30:40.285 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:40.287 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T09:30:40.411 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:40.413 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T09:30:40.423 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:40.425 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-09T09:30:40.552 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-09T09:30:40.554 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:40.554 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T09:30:40.680 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:40.682 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T09:30:40.692 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:40.693 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T09:30:40.808 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:40.810 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T09:30:40.830 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:40.832 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T09:30:40.945 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:40.947 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T09:30:40.948 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-09T09:30:40.977 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:40.979 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T09:30:41.092 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:41.094 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T09:30:41.114 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:41.115 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T09:30:41.230 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:41.232 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T09:30:41.252 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:41.254 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T09:30:41.369 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:41.371 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-09T09:30:41.391 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:41.393 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T09:30:41.496 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:41.498 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T09:30:41.539 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:41.541 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T09:30:41.633 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:41.639 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T09:30:41.665 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:41.667 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T09:30:41.770 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:41.772 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T09:30:41.802 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:41.804 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T09:30:41.908 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:41.911 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T09:30:41.914 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T09:30:41.941 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:41.943 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T09:30:42.111 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:42.112 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T09:30:42.123 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:42.124 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T09:30:42.270 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T09:30:42.282 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:42.283 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T09:30:42.409 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:42.411 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T09:30:42.432 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:42.433 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T09:30:42.559 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:42.561 INFO:teuthology.task.ansible.out:TASK [Remove revoked users] **************************************************** ok: [smithi173.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T09:30:42.583 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:42.585 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T09:30:42.701 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:42.703 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T09:30:42.723 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:42.724 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-09T09:30:42.830 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:42.832 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T09:30:42.883 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:42.885 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T09:30:42.978 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:42.979 INFO:teuthology.task.ansible.out:TASK [Remove revoked users] **************************************************** ok: [smithi173.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T09:30:43.010 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:43.012 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T09:30:43.115 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:43.117 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T09:30:43.147 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:43.149 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T09:30:43.263 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:43.265 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T09:30:43.287 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:43.289 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T09:30:43.408 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:43.410 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T09:30:43.419 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:43.421 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T09:30:43.537 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:43.539 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T09:30:43.560 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:43.561 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T09:30:43.678 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi173.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-09T09:30:43.689 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:43.691 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T09:30:43.815 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:43.817 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T09:30:43.826 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:43.828 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T09:30:43.951 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:43.953 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T09:30:43.963 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:43.965 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T09:30:44.089 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T09:30:44.090 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:44.091 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T09:30:44.217 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:44.219 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T09:30:44.229 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:44.230 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T09:30:44.347 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:44.349 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T09:30:44.370 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi173.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T09:30:44.485 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:44.487 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T09:30:44.507 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:44.509 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T09:30:44.623 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:44.625 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T09:30:44.627 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T09:30:44.645 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:44.647 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T09:30:44.760 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:44.761 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T09:30:44.782 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:44.784 INFO:teuthology.task.ansible.out:TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:44.785 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T09:30:44.840 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:44 +0000 (0:00:07.080) 0:00:56.541 ********* 2021-03-09T09:30:44.899 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:44.900 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T09:30:45.027 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T09:30:45.165 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T09:30:45.292 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T09:30:45.335 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-09T09:30:45.337 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282244.95398-1034-164910035595515/source", "state": "file", "uid": 0} 2021-03-09T09:30:45.363 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:45 +0000 (0:00:00.523) 0:00:57.064 ********* 2021-03-09T09:30:45.433 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:45.435 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} TASK [common : Log the OS name, version and release] *************************** ok: [smithi173.front.sepia.ceph.com] => { "msg": "Host smithi173.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-09T09:30:45.459 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:45 +0000 (0:00:00.096) 0:00:57.161 ********* 2021-03-09T09:30:45.539 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-09T09:30:45.541 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-09T09:30:45.564 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:45 +0000 (0:00:00.104) 0:00:57.265 ********* 2021-03-09T09:30:45.571 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:45.573 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T09:30:45.625 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:45.648 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:45 +0000 (0:00:00.084) 0:00:57.349 ********* 2021-03-09T09:30:45.650 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:45.686 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:45.688 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T09:30:45.718 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-09T09:30:45.720 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:45.742 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:45 +0000 (0:00:00.093) 0:00:57.443 ********* 2021-03-09T09:30:45.810 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-09T09:30:45.811 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:45.813 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T09:30:45.839 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:45 +0000 (0:00:00.097) 0:00:57.540 ********* 2021-03-09T09:30:45.841 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:45.909 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-09T09:30:45.911 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:45.932 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:45 +0000 (0:00:00.092) 0:00:57.633 ********* 2021-03-09T09:30:45.949 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:45.950 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T09:30:46.002 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:46.025 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.092) 0:00:57.726 ********* 2021-03-09T09:30:46.073 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:46.074 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T09:30:46.095 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:46.117 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.092) 0:00:57.819 ********* 2021-03-09T09:30:46.190 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-09T09:30:46.192 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T09:30:46.215 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.097) 0:00:57.917 ********* 2021-03-09T09:30:46.285 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-09T09:30:46.286 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:46.309 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.093) 0:00:58.010 ********* 2021-03-09T09:30:46.326 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:46.328 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T09:30:46.391 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-09T09:30:46.392 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi173.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:46.414 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.104) 0:00:58.115 ********* 2021-03-09T09:30:46.441 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:46.442 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T09:30:46.483 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:46.509 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.095) 0:00:58.210 ********* 2021-03-09T09:30:46.579 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:46.581 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T09:30:46.582 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:46.609 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.099) 0:00:58.310 ********* 2021-03-09T09:30:46.677 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-09T09:30:46.679 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:46.701 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.091) 0:00:58.402 ********* 2021-03-09T09:30:46.707 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:46.709 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T09:30:46.761 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:46.785 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.084) 0:00:58.487 ********* 2021-03-09T09:30:46.824 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T09:30:46.825 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options" 2021-03-09T09:30:46.827 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-09T09:30:46.856 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-09T09:30:46.858 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:46.880 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.094) 0:00:58.581 ********* 2021-03-09T09:30:46.950 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-09T09:30:46.951 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T09:30:46.977 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:46 +0000 (0:00:00.096) 0:00:58.678 ********* 2021-03-09T09:30:47.055 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.078) 0:00:58.756 ********* 2021-03-09T09:30:47.062 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-09T09:30:47.064 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.091 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.036) 0:00:58.793 ********* 2021-03-09T09:30:47.140 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:47.142 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.153 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.155 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.164 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.166 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.189 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.097) 0:00:58.890 ********* 2021-03-09T09:30:47.196 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.198 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.261 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T09:30:47.264 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.285 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.095) 0:00:58.986 ********* 2021-03-09T09:30:47.323 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:47.325 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2021-03-09T09:30:47.327 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.335 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.357 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi173.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:47.382 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.096) 0:00:59.083 ********* 2021-03-09T09:30:47.452 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T09:30:47.454 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.476 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.094) 0:00:59.177 ********* 2021-03-09T09:30:47.482 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:47.484 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-09T09:30:47.536 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.562 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.086) 0:00:59.263 ********* 2021-03-09T09:30:47.579 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:47.581 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-09T09:30:47.591 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.592 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.602 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.613 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.615 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.624 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.626 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.649 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.086) 0:00:59.350 ********* 2021-03-09T09:30:47.656 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.657 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.658 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.669 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.670 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.681 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.688 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.692 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.704 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.706 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.716 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.717 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-09T09:30:47.717 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.718 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.741 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.092) 0:00:59.442 ********* 2021-03-09T09:30:47.748 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.750 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.752 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.763 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.765 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.775 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.777 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.786 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.798 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.799 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.809 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-09T09:30:47.811 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.838 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.096) 0:00:59.539 ********* 2021-03-09T09:30:47.847 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.849 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.849 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.850 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.852 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.862 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.864 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.874 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.876 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.886 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.897 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.908 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.910 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.934 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:47 +0000 (0:00:00.096) 0:00:59.635 ********* 2021-03-09T09:30:47.941 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:47.942 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.944 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.944 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.945 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.945 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.947 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.957 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.968 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.979 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:47.990 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.001 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.002 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.002 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi173.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:48.026 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.092) 0:00:59.728 ********* 2021-03-09T09:30:48.034 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:48.036 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.036 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.036 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": 2021-03-09T09:30:48.038 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.048 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.049 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.061 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.063 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.073 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.074 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.084 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.095 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.097 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.126 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.099) 0:00:59.827 ********* 2021-03-09T09:30:48.133 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.135 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.137 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.148 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.149 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.159 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.161 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.171 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.173 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.182 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.193 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.195 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.218 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.091) 0:00:59.919 ********* 2021-03-09T09:30:48.225 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.227 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.229 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.240 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.242 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.252 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.253 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.263 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.265 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.275 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.276 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.286 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.288 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.310 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.092) 0:01:00.011 ********* 2021-03-09T09:30:48.317 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:48.319 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.321 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.331 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.332 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.332 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.343 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.345 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.355 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.356 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.366 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.377 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.379 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.388 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.389 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-09T09:30:48.389 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.390 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.390 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-09T09:30:48.391 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.417 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.106) 0:01:00.118 ********* 2021-03-09T09:30:48.423 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T09:30:48.425 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.426 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.428 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.439 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.441 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.491 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.073) 0:01:00.192 ********* 2021-03-09T09:30:48.498 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-09T09:30:48.499 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.525 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.034) 0:01:00.226 ********* 2021-03-09T09:30:48.563 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-09T09:30:48.565 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.588 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.062) 0:01:00.289 ********* 2021-03-09T09:30:48.594 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T09:30:48.596 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.620 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.032) 0:01:00.321 ********* 2021-03-09T09:30:48.805 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:48.808 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} TASK [common : Get the current timezone] *************************************** ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.011879", "end": "2021-03-09 09:30:48.921304", "rc": 0, "start": "2021-03-09 09:30:48.909425", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-09T09:30:48.831 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.211) 0:01:00.533 ********* 2021-03-09T09:30:48.910 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-09T09:30:48.912 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:48.933 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:48 +0000 (0:00:00.101) 0:01:00.635 ********* 2021-03-09T09:30:48.950 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:48.952 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-09T09:30:49.004 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-09T09:30:49.006 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:49.039 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:49 +0000 (0:00:00.105) 0:01:00.740 ********* 2021-03-09T09:30:49.090 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:49.092 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T09:30:49.123 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T09:30:49.125 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:49.134 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:49.136 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:49.158 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:49 +0000 (0:00:00.119) 0:01:00.859 ********* 2021-03-09T09:30:49.227 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:49.229 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} TASK [common : Get the current timezone] *************************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:49.251 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:49 +0000 (0:00:00.093) 0:01:00.953 ********* 2021-03-09T09:30:49.319 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-09T09:30:49.321 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:49.343 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:49 +0000 (0:00:00.091) 0:01:01.044 ********* 2021-03-09T09:30:49.360 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:49.362 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T09:30:49.413 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:49.439 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:49 +0000 (0:00:00.095) 0:01:01.140 ********* 2021-03-09T09:30:49.497 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:49.499 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T09:30:49.510 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-09T09:30:49.512 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:49.533 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:49 +0000 (0:00:00.094) 0:01:01.235 ********* 2021-03-09T09:30:49.623 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-09T09:30:49.625 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:49.648 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:49 +0000 (0:00:00.114) 0:01:01.349 ********* 2021-03-09T09:30:49.655 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:49.657 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T09:30:49.771 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T09:30:49.907 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T09:30:50.043 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T09:30:50.180 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T09:30:50.317 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T09:30:50.391 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-09T09:30:50.392 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.613213", "end": "2021-03-09 09:30:50.500416", "rc": 0, "start": "2021-03-09 09:30:49.887203", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:30:50.415 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:50 +0000 (0:00:00.766) 0:01:02.116 ********* 2021-03-09T09:30:50.453 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:50.455 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T09:30:50.485 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-09T09:30:50.488 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:50.508 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:50 +0000 (0:00:00.093) 0:01:02.210 ********* 2021-03-09T09:30:50.588 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:50.590 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T09:30:50.704 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-09T09:30:50.706 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.022362", "end": "2021-03-09 09:30:50.818053", "rc": 0, "start": "2021-03-09 09:30:50.795691", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:30:50.729 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:50 +0000 (0:00:00.220) 0:01:02.430 ********* 2021-03-09T09:30:50.735 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:50.737 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T09:30:50.863 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T09:30:51.002 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T09:30:51.143 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T09:30:51.278 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T09:30:51.414 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T09:30:51.561 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T09:30:51.697 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T09:30:51.835 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T09:30:51.971 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T09:30:52.111 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T09:30:52.247 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T09:30:52.384 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T09:30:52.520 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T09:30:52.655 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T09:30:52.657 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:52.791 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T09:30:52.938 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T09:30:53.074 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T09:30:53.252 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T09:30:53.399 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T09:30:53.566 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T09:30:53.568 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:53.706 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T09:30:53.843 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T09:30:53.864 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-09T09:30:53.866 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:30:53.888 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:53 +0000 (0:00:03.158) 0:01:05.589 ********* 2021-03-09T09:30:53.977 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:53.979 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T09:30:54.124 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T09:30:54.261 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T09:30:54.396 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-09T09:30:54.531 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T09:30:54.667 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T09:30:54.792 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-09T09:30:54.793 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:30:54.815 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:54 +0000 (0:00:00.926) 0:01:06.516 ********* 2021-03-09T09:30:54.823 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:54.825 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T09:30:54.887 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:54.910 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:54 +0000 (0:00:00.095) 0:01:06.611 ********* 2021-03-09T09:30:54.958 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T09:30:54.960 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T09:30:55.085 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T09:30:55.232 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T09:30:55.369 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T09:30:55.507 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T09:30:55.558 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:55 +0000 (0:00:00.647) 0:01:07.259 ********* 2021-03-09T09:30:55.566 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-09T09:30:55.567 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-09T09:30:55.593 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:55 +0000 (0:00:00.034) 0:01:07.294 ********* 2021-03-09T09:30:55.953 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-09T09:30:55.955 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282255.6386385-1505-159779419299695/source", "state": "file", "uid": 0} 2021-03-09T09:30:55.978 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:55 +0000 (0:00:00.385) 0:01:07.679 ********* 2021-03-09T09:30:56.047 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-09T09:30:56.049 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => { "msg": "Host smithi143.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-09T09:30:56.080 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.101) 0:01:07.781 ********* 2021-03-09T09:30:56.161 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-09T09:30:56.163 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-09T09:30:56.185 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.105) 0:01:07.886 ********* 2021-03-09T09:30:56.254 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:56.277 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.091) 0:01:07.978 ********* 2021-03-09T09:30:56.347 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-09T09:30:56.348 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:56.371 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.094) 0:01:08.073 ********* 2021-03-09T09:30:56.439 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-09T09:30:56.441 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:56.462 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.090) 0:01:08.164 ********* 2021-03-09T09:30:56.530 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-09T09:30:56.532 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:56.553 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.091) 0:01:08.255 ********* 2021-03-09T09:30:56.570 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-09T09:30:56.572 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:30:56.595 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.041) 0:01:08.296 ********* 2021-03-09T09:30:56.613 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-09T09:30:56.615 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:56.637 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.042) 0:01:08.339 ********* 2021-03-09T09:30:56.707 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-09T09:30:56.708 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:56.730 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.092) 0:01:08.431 ********* 2021-03-09T09:30:56.797 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-09T09:30:56.799 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:56.822 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.092) 0:01:08.523 ********* 2021-03-09T09:30:56.891 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:56.915 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:56 +0000 (0:00:00.092) 0:01:08.616 ********* 2021-03-09T09:30:56.985 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T09:30:56.986 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T09:30:56.997 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-09T09:30:56.998 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi143.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:57.020 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.105) 0:01:08.721 ********* 2021-03-09T09:30:57.091 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-09T09:30:57.093 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:57.115 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.094) 0:01:08.816 ********* 2021-03-09T09:30:57.184 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-09T09:30:57.185 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:57.207 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.091) 0:01:08.908 ********* 2021-03-09T09:30:57.276 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-09T09:30:57.277 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:57.305 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.097) 0:01:09.006 ********* 2021-03-09T09:30:57.312 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T09:30:57.312 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T09:30:57.377 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:57.403 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.099) 0:01:09.105 ********* 2021-03-09T09:30:57.473 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:57.474 INFO:teuthology.task.ansible.out:TASK [common : Register with subscription-manager.] **************************** skipping: [smithi143.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:57.500 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.096) 0:01:09.201 ********* 2021-03-09T09:30:57.571 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-09T09:30:57.573 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi173.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-09T09:30:57.599 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.098) 0:01:09.300 ********* 2021-03-09T09:30:57.622 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.023) 0:01:09.323 ********* 2021-03-09T09:30:57.663 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-09T09:30:57.666 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:57.692 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.069) 0:01:09.393 ********* 2021-03-09T09:30:57.770 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-09T09:30:57.772 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:30:57.794 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.102) 0:01:09.495 ********* 2021-03-09T09:30:57.815 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.020) 0:01:09.516 ********* 2021-03-09T09:30:57.874 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T09:30:57.877 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:57.898 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.083) 0:01:09.600 ********* 2021-03-09T09:30:57.979 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-09T09:30:57.981 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:58.002 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:57 +0000 (0:00:00.103) 0:01:09.703 ********* 2021-03-09T09:30:58.075 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T09:30:58.077 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:58.099 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.097) 0:01:09.800 ********* 2021-03-09T09:30:58.180 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-09T09:30:58.181 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:58.204 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.104) 0:01:09.905 ********* 2021-03-09T09:30:58.210 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T09:30:58.212 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-09T09:30:58.275 INFO:teuthology.task.ansible.out: 2021-03-09T09:30:58.277 INFO:teuthology.task.ansible.out:TASK [common : Store list of enabled repos] ************************************ skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:58.299 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.095) 0:01:10.000 ********* 2021-03-09T09:30:58.369 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-09T09:30:58.371 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:58.396 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.097) 0:01:10.097 ********* 2021-03-09T09:30:58.466 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-09T09:30:58.468 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:58.490 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.093) 0:01:10.191 ********* 2021-03-09T09:30:58.499 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T09:30:58.501 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-09T09:30:58.566 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-09T09:30:58.568 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:58.602 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.111) 0:01:10.303 ********* 2021-03-09T09:30:58.674 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-09T09:30:58.676 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:58.697 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.095) 0:01:10.398 ********* 2021-03-09T09:30:58.766 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-09T09:30:58.768 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:58.789 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.092) 0:01:10.490 ********* 2021-03-09T09:30:58.796 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T09:30:58.798 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-09T09:30:58.825 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.035) 0:01:10.526 ********* 2021-03-09T09:30:58.853 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-09T09:30:58.855 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:58.879 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.053) 0:01:10.580 ********* 2021-03-09T09:30:58.907 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T09:30:58.909 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-09T09:30:58.932 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.053) 0:01:10.634 ********* 2021-03-09T09:30:58.960 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-09T09:30:58.962 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:58.984 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:58 +0000 (0:00:00.051) 0:01:10.686 ********* 2021-03-09T09:30:59.002 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-09T09:30:59.004 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.028 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.043) 0:01:10.729 ********* 2021-03-09T09:30:59.067 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-09T09:30:59.069 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.093 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.064) 0:01:10.794 ********* 2021-03-09T09:30:59.110 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T09:30:59.112 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:30:59.137 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.044) 0:01:10.838 ********* 2021-03-09T09:30:59.176 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-09T09:30:59.178 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.206 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.068) 0:01:10.907 ********* 2021-03-09T09:30:59.234 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T09:30:59.236 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.246 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.248 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.271 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.065) 0:01:10.972 ********* 2021-03-09T09:30:59.277 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T09:30:59.279 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.304 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.032) 0:01:11.005 ********* 2021-03-09T09:30:59.342 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-09T09:30:59.344 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.367 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.062) 0:01:11.068 ********* 2021-03-09T09:30:59.501 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T09:30:59.503 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.016323", "end": "2021-03-09 09:30:59.586130", "rc": 0, "start": "2021-03-09 09:30:59.569807", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-09T09:30:59.526 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.159) 0:01:11.228 ********* 2021-03-09T09:30:59.606 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-09T09:30:59.608 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.631 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.104) 0:01:11.332 ********* 2021-03-09T09:30:59.710 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-09T09:30:59.712 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.734 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.102) 0:01:11.435 ********* 2021-03-09T09:30:59.826 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T09:30:59.828 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.838 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.840 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.875 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.141) 0:01:11.576 ********* 2021-03-09T09:30:59.948 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T09:30:59.950 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:30:59.971 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:30:59 +0000 (0:00:00.096) 0:01:11.672 ********* 2021-03-09T09:31:00.040 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-09T09:31:00.042 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:00.063 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:00 +0000 (0:00:00.091) 0:01:11.764 ********* 2021-03-09T09:31:00.131 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-09T09:31:00.133 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:00.154 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:00 +0000 (0:00:00.090) 0:01:11.855 ********* 2021-03-09T09:31:00.223 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-09T09:31:00.225 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:00.246 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:00 +0000 (0:00:00.092) 0:01:11.947 ********* 2021-03-09T09:31:00.325 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-09T09:31:00.327 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:31:00.353 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:00 +0000 (0:00:00.107) 0:01:12.054 ********* 2021-03-09T09:31:00.360 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-09T09:31:00.361 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:00.385 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:00 +0000 (0:00:00.032) 0:01:12.087 ********* 2021-03-09T09:31:00.705 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-09T09:31:00.707 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-09T09:31:00.730 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:00 +0000 (0:00:00.344) 0:01:12.431 ********* 2021-03-09T09:31:01.081 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-09T09:31:01.083 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-09T09:31:01.105 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:01 +0000 (0:00:00.375) 0:01:12.807 ********* 2021-03-09T09:31:01.310 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T09:31:01.311 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-09T09:31:01.415 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-09T09:31:01.417 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:01.418 INFO:teuthology.task.ansible.out:TASK [common : Set the hardware clock] ***************************************** changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.887455", "end": "2021-03-09 09:31:01.500439", "rc": 0, "start": "2021-03-09 09:31:00.612984", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:31:01.441 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:01 +0000 (0:00:00.335) 0:01:13.142 ********* 2021-03-09T09:31:01.462 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:01 +0000 (0:00:00.021) 0:01:13.163 ********* 2021-03-09T09:31:01.511 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-09T09:31:01.512 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:01.534 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:01 +0000 (0:00:00.072) 0:01:13.236 ********* 2021-03-09T09:31:01.541 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-09T09:31:01.543 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:01.566 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:01 +0000 (0:00:00.031) 0:01:13.268 ********* 2021-03-09T09:31:01.646 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-09T09:31:01.648 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:31:01.670 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:01 +0000 (0:00:00.103) 0:01:13.371 ********* 2021-03-09T09:31:01.739 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-09T09:31:01.741 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023159", "end": "2021-03-09 09:31:01.817755", "rc": 0, "start": "2021-03-09 09:31:01.794596", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:31:01.766 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:01 +0000 (0:00:00.096) 0:01:13.467 ********* 2021-03-09T09:31:01.773 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-09T09:31:01.774 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:01.804 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:01 +0000 (0:00:00.038) 0:01:13.505 ********* 2021-03-09T09:31:02.195 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-09T09:31:02.197 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-09T09:31:02.220 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:02 +0000 (0:00:00.415) 0:01:13.921 ********* 2021-03-09T09:31:02.881 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-09T09:31:02.882 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:28:37 UTC", "ActiveEnterTimestampMonotonic": "17798769", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target systemd-tmpfiles-setup.service -.mount system.slice systemd-journald.socket var-run.mount network.target sysinit.target remote-fs.target local-fs.target nss-lookup.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:28:37 UTC", "AssertTimestampMonotonic": "17798120", "Before": "multi-user.target xdm.service plymouth-quit.service shutdown.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:28:37 UTC", "ConditionTimestampMonotonic": "17798119", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-09T09:31:02.884 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1194", "ExecMainStartTimestamp": "Tue 2021-03-09 09:28:37 UTC", "ExecMainStartTimestampMonotonic": "17798744", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:28:37 UTC", "InactiveExitTimestampMonotonic": "17798769", "InvocationID": "54800a6831ea4f58b2f24235227d23d2", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2021-03-09T09:31:02.885 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1194", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2021-03-09T09:31:02.887 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:28:37 UTC", "StateChangeTimestampMonotonic": "17798769", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:28:37 UTC", "WatchdogTimestampMonotonic": "17798769", "WatchdogUSec": "0"}} 2021-03-09T09:31:02.906 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:02 +0000 (0:00:00.686) 0:01:14.607 ********* 2021-03-09T09:31:03.099 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-09T09:31:03.100 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.123 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.216) 0:01:14.824 ********* 2021-03-09T09:31:03.192 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-09T09:31:03.194 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.217 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.093) 0:01:14.918 ********* 2021-03-09T09:31:03.285 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-09T09:31:03.287 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.311 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.094) 0:01:15.012 ********* 2021-03-09T09:31:03.380 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-09T09:31:03.382 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.405 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.094) 0:01:15.106 ********* 2021-03-09T09:31:03.474 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-09T09:31:03.476 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.500 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.095) 0:01:15.202 ********* 2021-03-09T09:31:03.569 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-09T09:31:03.571 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.593 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.092) 0:01:15.294 ********* 2021-03-09T09:31:03.661 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-09T09:31:03.663 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.685 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.091) 0:01:15.386 ********* 2021-03-09T09:31:03.754 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-09T09:31:03.756 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.777 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.092) 0:01:15.478 ********* 2021-03-09T09:31:03.859 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-09T09:31:03.861 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.882 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.105) 0:01:15.583 ********* 2021-03-09T09:31:03.951 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:03.975 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:03 +0000 (0:00:00.092) 0:01:15.676 ********* 2021-03-09T09:31:04.053 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-09T09:31:04.055 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-09T09:31:04.078 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:04 +0000 (0:00:00.103) 0:01:15.780 ********* 2021-03-09T09:31:04.284 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-09T09:31:04.285 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-09T09:31:04.308 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:04 +0000 (0:00:00.229) 0:01:16.009 ********* 2021-03-09T09:31:04.513 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-09T09:31:04.515 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-09T09:31:04.537 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:04 +0000 (0:00:00.228) 0:01:16.238 ********* 2021-03-09T09:31:04.658 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-09T09:31:04.660 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:04.683 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:04 +0000 (0:00:00.145) 0:01:16.384 ********* 2021-03-09T09:31:04.732 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-09T09:31:04.733 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T09:31:04.759 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:04 +0000 (0:00:00.076) 0:01:16.461 ********* 2021-03-09T09:31:04.902 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-09T09:31:04.904 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T09:31:05.018 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:05 +0000 (0:00:00.258) 0:01:16.719 ********* 2021-03-09T09:31:05.201 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-09T09:31:05.203 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T09:31:05.224 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:05 +0000 (0:00:00.205) 0:01:16.925 ********* 2021-03-09T09:31:05.293 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-09T09:31:05.294 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:05.316 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:05 +0000 (0:00:00.092) 0:01:17.017 ********* 2021-03-09T09:31:05.384 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-09T09:31:05.386 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:05.407 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:05 +0000 (0:00:00.091) 0:01:17.108 ********* 2021-03-09T09:31:05.609 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-09T09:31:05.611 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-09T09:31:05.633 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:05 +0000 (0:00:00.225) 0:01:17.334 ********* 2021-03-09T09:31:05.733 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-09T09:31:05.734 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:05.757 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:05 +0000 (0:00:00.124) 0:01:17.458 ********* 2021-03-09T09:31:05.826 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-09T09:31:05.832 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:05.850 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:05 +0000 (0:00:00.092) 0:01:17.551 ********* 2021-03-09T09:31:06.086 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-09T09:31:06.087 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-09T09:31:06.111 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:06 +0000 (0:00:00.260) 0:01:17.812 ********* 2021-03-09T09:31:06.190 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-09T09:31:06.192 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi173"}, "changed": false} 2021-03-09T09:31:06.214 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:06 +0000 (0:00:00.103) 0:01:17.915 ********* 2021-03-09T09:31:06.220 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-09T09:31:06.222 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-09T09:31:06.249 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:06 +0000 (0:00:00.034) 0:01:17.950 ********* 2021-03-09T09:31:07.077 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-09T09:31:07.079 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi173", "ansible_hostname": "smithi173", "ansible_nodename": "smithi173"}, "changed": false, "name": "smithi173"} 2021-03-09T09:31:07.102 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:07 +0000 (0:00:00.853) 0:01:18.803 ********* 2021-03-09T09:31:07.140 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-09T09:31:07.142 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:07.165 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:07 +0000 (0:00:00.062) 0:01:18.866 ********* 2021-03-09T09:31:07.474 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-09T09:31:07.476 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-09T09:31:07.500 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:07 +0000 (0:00:00.335) 0:01:19.201 ********* 2021-03-09T09:31:07.538 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T09:31:07.540 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T09:31:07.832 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T09:31:07.885 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-09T09:31:07.887 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-09T09:31:07.909 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:07 +0000 (0:00:00.408) 0:01:19.610 ********* 2021-03-09T09:31:08.134 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T09:31:08.136 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-09T09:31:08.163 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:08 +0000 (0:00:00.254) 0:01:19.864 ********* 2021-03-09T09:31:08.294 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-09T09:31:08.296 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:31:08.318 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:08 +0000 (0:00:00.155) 0:01:20.019 ********* 2021-03-09T09:31:08.355 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-09T09:31:08.357 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T09:31:08.382 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:08 +0000 (0:00:00.064) 0:01:20.083 ********* 2021-03-09T09:31:08.452 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:08.476 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:08 +0000 (0:00:00.093) 0:01:20.177 ********* 2021-03-09T09:31:08.555 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T09:31:08.557 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:08.579 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:08 +0000 (0:00:00.103) 0:01:20.280 ********* 2021-03-09T09:31:08.648 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T09:31:08.649 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:08.671 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:08 +0000 (0:00:00.092) 0:01:20.372 ********* 2021-03-09T09:31:08.689 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T09:31:08.691 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-09T09:31:08.754 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:08.789 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:08 +0000 (0:00:00.117) 0:01:20.490 ********* 2021-03-09T09:31:08.860 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T09:31:08.862 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:08.883 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:08 +0000 (0:00:00.094) 0:01:20.585 ********* 2021-03-09T09:31:08.963 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T09:31:08.964 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:08.988 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:08 +0000 (0:00:00.104) 0:01:20.689 ********* 2021-03-09T09:31:08.995 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T09:31:08.997 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-09T09:31:09.060 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T09:31:09.062 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.085 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.097) 0:01:20.786 ********* 2021-03-09T09:31:09.164 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-09T09:31:09.166 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.188 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.103) 0:01:20.889 ********* 2021-03-09T09:31:09.257 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-09T09:31:09.259 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi143.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-09T09:31:09.281 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.093) 0:01:20.982 ********* 2021-03-09T09:31:09.315 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.032) 0:01:21.015 ********* 2021-03-09T09:31:09.352 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-09T09:31:09.376 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.061) 0:01:21.077 ********* 2021-03-09T09:31:09.393 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T09:31:09.395 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-09T09:31:09.435 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.058) 0:01:21.136 ********* 2021-03-09T09:31:09.587 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.047) 0:01:21.183 ********* TASK [common : Include yum_systems vars] *************************************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.066) 0:01:21.250 ********* TASK [testnode : Install GPG] ************************************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.607 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.057) 0:01:21.308 ********* 2021-03-09T09:31:09.638 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T09:31:09.640 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:31:09.665 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.058) 0:01:21.366 ********* 2021-03-09T09:31:09.696 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-09T09:31:09.697 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.699 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.701 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:09.726 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.060) 0:01:21.427 ********* 2021-03-09T09:31:09.766 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T09:31:09.768 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.769 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.780 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.806 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.080) 0:01:21.508 ********* 2021-03-09T09:31:09.835 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-09T09:31:09.838 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.872 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.065) 0:01:21.573 ********* 2021-03-09T09:31:09.890 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.915 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.043) 0:01:21.616 ********* 2021-03-09T09:31:09.953 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:31:09.955 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:09.979 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:09 +0000 (0:00:00.063) 0:01:21.680 ********* 2021-03-09T09:31:10.059 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-09T09:31:10.061 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.083 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.104) 0:01:21.784 ********* 2021-03-09T09:31:10.152 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-09T09:31:10.154 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.175 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.092) 0:01:21.877 ********* 2021-03-09T09:31:10.244 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-09T09:31:10.246 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.271 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.095) 0:01:21.972 ********* 2021-03-09T09:31:10.340 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-09T09:31:10.342 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.363 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.092) 0:01:22.065 ********* 2021-03-09T09:31:10.432 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T09:31:10.434 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.455 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.091) 0:01:22.156 ********* 2021-03-09T09:31:10.524 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T09:31:10.526 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.547 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.091) 0:01:22.248 ********* 2021-03-09T09:31:10.626 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:31:10.628 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.650 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.102) 0:01:22.351 ********* 2021-03-09T09:31:10.730 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-09T09:31:10.732 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.754 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.103) 0:01:22.455 ********* 2021-03-09T09:31:10.834 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T09:31:10.836 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.858 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.104) 0:01:22.559 ********* 2021-03-09T09:31:10.865 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-09T09:31:10.867 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:10.892 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.033) 0:01:22.593 ********* 2021-03-09T09:31:10.930 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:31:10.932 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:10.957 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:10 +0000 (0:00:00.064) 0:01:22.658 ********* 2021-03-09T09:31:11.036 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-09T09:31:11.038 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:11.059 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.102) 0:01:22.760 ********* 2021-03-09T09:31:11.129 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-09T09:31:11.131 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:11.159 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.100) 0:01:22.860 ********* 2021-03-09T09:31:11.238 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-09T09:31:11.240 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:11.262 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.102) 0:01:22.963 ********* 2021-03-09T09:31:11.269 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-09T09:31:11.270 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-09T09:31:11.299 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.037) 0:01:23.000 ********* 2021-03-09T09:31:11.327 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-09T09:31:11.329 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:11.352 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.052) 0:01:23.053 ********* 2021-03-09T09:31:11.423 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-09T09:31:11.425 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:11.446 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.094) 0:01:23.147 ********* 2021-03-09T09:31:11.515 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-09T09:31:11.517 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:11.539 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.093) 0:01:23.240 ********* 2021-03-09T09:31:11.621 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-09T09:31:11.623 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:11.647 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.107) 0:01:23.348 ********* 2021-03-09T09:31:11.729 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-09T09:31:11.731 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:11.753 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.106) 0:01:23.454 ********* 2021-03-09T09:31:11.762 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-09T09:31:11.764 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-09T09:31:11.789 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.036) 0:01:23.490 ********* 2021-03-09T09:31:11.945 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-09T09:31:11.947 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004870", "end": "2021-03-09 09:31:12.055734", "rc": 0, "start": "2021-03-09 09:31:12.050864", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:31:11.970 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:11 +0000 (0:00:00.180) 0:01:23.671 ********* 2021-03-09T09:31:11.988 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T09:31:11.990 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-09T09:31:12.042 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-09T09:31:12.071 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:12 +0000 (0:00:00.101) 0:01:23.773 ********* 2021-03-09T09:31:12.095 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T09:31:12.097 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-09T09:31:12.123 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:12 +0000 (0:00:00.051) 0:01:23.824 ********* 2021-03-09T09:31:12.202 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-09T09:31:12.204 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:12.226 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:12 +0000 (0:00:00.103) 0:01:23.928 ********* 2021-03-09T09:31:12.295 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-09T09:31:12.297 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T09:31:12.319 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:12 +0000 (0:00:00.092) 0:01:24.020 ********* 2021-03-09T09:31:12.399 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-09T09:31:12.401 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:12.426 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:12 +0000 (0:00:00.106) 0:01:24.127 ********* 2021-03-09T09:31:12.485 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-09T09:31:12.487 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-09T09:31:12.510 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:12 +0000 (0:00:00.084) 0:01:24.211 ********* 2021-03-09T09:31:12.590 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-09T09:31:12.592 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:12.613 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:12 +0000 (0:00:00.103) 0:01:24.315 ********* 2021-03-09T09:31:12.802 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-09T09:31:12.804 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-09T09:31:12.827 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:12 +0000 (0:00:00.213) 0:01:24.528 ********* 2021-03-09T09:31:13.107 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-09T09:31:13.108 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:28:52 UTC", "ActiveEnterTimestampMonotonic": "21643700", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice local-fs.target network.target time-sync.target -.mount basic.target sysinit.target systemd-tmpfiles-setup.service remote-fs.target var-run.mount nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:28:52 UTC", "AssertTimestampMonotonic": "21642451", "Before": "plymouth-quit.service shutdown.target multi-user.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:28:52 UTC", "ConditionTimestampMonotonic": "21642451", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-09T09:31:13.110 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1200", "ExecMainStartTimestamp": "Tue 2021-03-09 09:28:52 UTC", "ExecMainStartTimestampMonotonic": "21643664", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:28:52 UTC", "InactiveExitTimestampMonotonic": "21643700", "InvocationID": "db422130cc114fa2b12a6ce3257dc36e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2021-03-09T09:31:13.112 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1200", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2021-03-09T09:31:13.113 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:28:52 UTC", "StateChangeTimestampMonotonic": "21643700", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:28:52 UTC", "WatchdogTimestampMonotonic": "21643699", "WatchdogUSec": "0"}} 2021-03-09T09:31:13.132 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:13 +0000 (0:00:00.304) 0:01:24.833 ********* 2021-03-09T09:31:13.212 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:13.235 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:13 +0000 (0:00:00.102) 0:01:24.936 ********* 2021-03-09T09:31:13.340 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-09T09:31:13.342 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:13.363 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:13 +0000 (0:00:00.128) 0:01:25.064 ********* 2021-03-09T09:31:13.442 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-09T09:31:13.444 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:13.465 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:13 +0000 (0:00:00.102) 0:01:25.166 ********* 2021-03-09T09:31:13.565 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-09T09:31:13.567 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:13.588 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:13 +0000 (0:00:00.122) 0:01:25.289 ********* 2021-03-09T09:31:13.667 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-09T09:31:13.669 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:13.691 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:13 +0000 (0:00:00.103) 0:01:25.392 ********* 2021-03-09T09:31:13.770 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-09T09:31:13.772 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:13.798 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:13 +0000 (0:00:00.106) 0:01:25.499 ********* 2021-03-09T09:31:13.867 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-09T09:31:13.869 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:13.896 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:13 +0000 (0:00:00.092) 0:01:25.592 ********* 2021-03-09T09:31:13.935 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T09:31:13.937 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-09T09:31:13.968 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-09T09:31:13.970 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:13.992 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:13 +0000 (0:00:00.100) 0:01:25.693 ********* 2021-03-09T09:31:14.065 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-09T09:31:14.066 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:14.097 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:14 +0000 (0:00:00.105) 0:01:25.799 ********* 2021-03-09T09:31:14.190 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:14.214 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:14 +0000 (0:00:00.116) 0:01:25.915 ********* 2021-03-09T09:31:14.299 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-09T09:31:14.301 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-09T09:31:14.323 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:14 +0000 (0:00:00.109) 0:01:26.025 ********* 2021-03-09T09:31:14.414 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-09T09:31:14.416 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-09T09:31:14.439 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:14 +0000 (0:00:00.115) 0:01:26.140 ********* 2021-03-09T09:31:14.519 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-09T09:31:14.521 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-09T09:31:14.544 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:14 +0000 (0:00:00.104) 0:01:26.245 ********* 2021-03-09T09:31:14.647 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-09T09:31:14.648 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T09:31:14.670 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:14 +0000 (0:00:00.126) 0:01:26.371 ********* 2021-03-09T09:31:14.886 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-09T09:31:14.888 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-09T09:31:14.912 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:14 +0000 (0:00:00.241) 0:01:26.613 ********* 2021-03-09T09:31:14.921 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T09:31:14.923 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-09T09:31:14.949 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:14 +0000 (0:00:00.037) 0:01:26.650 ********* 2021-03-09T09:31:15.127 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-09T09:31:15.128 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T09:31:15.150 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:15 +0000 (0:00:00.201) 0:01:26.852 ********* 2021-03-09T09:31:15.152 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:15.245 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-09T09:31:15.247 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:15.273 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:15 +0000 (0:00:00.122) 0:01:26.974 ********* 2021-03-09T09:31:15.363 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-09T09:31:15.365 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:15.389 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:15 +0000 (0:00:00.115) 0:01:27.090 ********* 2021-03-09T09:31:15.608 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-09T09:31:15.610 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-09T09:31:15.632 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:15 +0000 (0:00:00.243) 0:01:27.333 ********* 2021-03-09T09:31:15.934 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-09T09:31:15.936 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:31:15.958 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:15 +0000 (0:00:00.326) 0:01:27.660 ********* 2021-03-09T09:31:16.029 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-09T09:31:16.053 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:16 +0000 (0:00:00.095) 0:01:27.755 ********* 2021-03-09T09:31:16.091 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-09T09:31:16.093 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-09T09:31:16.117 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:16 +0000 (0:00:00.063) 0:01:27.818 ********* 2021-03-09T09:31:16.209 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-09T09:31:16.211 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi143"}, "changed": false} 2021-03-09T09:31:16.233 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:16 +0000 (0:00:00.116) 0:01:27.935 ********* 2021-03-09T09:31:16.271 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-09T09:31:16.274 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036887", "end": "2021-03-09 09:31:16.382367", "rc": 0, "start": "2021-03-09 09:31:16.345480", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:31:16.298 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:16 +0000 (0:00:00.064) 0:01:27.999 ********* 2021-03-09T09:31:16.789 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-09T09:31:16.791 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi143", "ansible_hostname": "smithi143", "ansible_nodename": "smithi143"}, "changed": false, "name": "smithi143"} 2021-03-09T09:31:16.813 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:16 +0000 (0:00:00.515) 0:01:28.514 ********* 2021-03-09T09:31:17.187 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-09T09:31:17.189 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-09T09:31:17.214 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:17 +0000 (0:00:00.401) 0:01:28.915 ********* 2021-03-09T09:31:17.595 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-09T09:31:17.597 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-09T09:31:17.618 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:17 +0000 (0:00:00.403) 0:01:29.319 ********* 2021-03-09T09:31:18.045 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-09T09:31:18.047 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T09:31:18.068 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.449) 0:01:29.769 ********* 2021-03-09T09:31:18.137 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-09T09:31:18.139 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:18.160 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.092) 0:01:29.862 ********* 2021-03-09T09:31:18.230 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T09:31:18.232 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:18.254 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.093) 0:01:29.955 ********* 2021-03-09T09:31:18.323 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T09:31:18.324 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:18.346 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.092) 0:01:30.047 ********* 2021-03-09T09:31:18.415 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T09:31:18.417 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:18.438 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.092) 0:01:30.140 ********* 2021-03-09T09:31:18.507 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T09:31:18.509 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:18.530 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.091) 0:01:30.231 ********* 2021-03-09T09:31:18.600 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T09:31:18.601 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:18.622 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.092) 0:01:30.324 ********* 2021-03-09T09:31:18.691 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T09:31:18.693 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:18.715 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.092) 0:01:30.416 ********* 2021-03-09T09:31:18.806 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-09T09:31:18.808 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:18.832 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.117) 0:01:30.534 ********* 2021-03-09T09:31:18.912 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-09T09:31:18.935 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:18 +0000 (0:00:00.102) 0:01:30.637 ********* 2021-03-09T09:31:19.005 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-09T09:31:19.030 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.094) 0:01:30.731 ********* 2021-03-09T09:31:19.100 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-09T09:31:19.123 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.092) 0:01:30.824 ********* 2021-03-09T09:31:19.151 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-09T09:31:19.153 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:31:19.178 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.055) 0:01:30.880 ********* 2021-03-09T09:31:19.197 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-09T09:31:19.198 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:19.222 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.043) 0:01:30.923 ********* 2021-03-09T09:31:19.302 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-09T09:31:19.304 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:19.313 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:19.338 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.115) 0:01:31.039 ********* 2021-03-09T09:31:19.406 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-09T09:31:19.408 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:19.430 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.092) 0:01:31.131 ********* 2021-03-09T09:31:19.499 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:31:19.501 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:19.522 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.092) 0:01:31.223 ********* 2021-03-09T09:31:19.602 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-09T09:31:19.604 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:19.625 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.102) 0:01:31.326 ********* 2021-03-09T09:31:19.694 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-09T09:31:19.696 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:19.717 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.092) 0:01:31.418 ********* 2021-03-09T09:31:19.786 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-09T09:31:19.788 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:19.810 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.092) 0:01:31.511 ********* 2021-03-09T09:31:19.889 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:19.891 INFO:teuthology.task.ansible.out:TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:19.918 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:19 +0000 (0:00:00.108) 0:01:31.619 ********* 2021-03-09T09:31:19.990 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T09:31:19.991 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.014 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.095) 0:01:31.715 ********* 2021-03-09T09:31:20.083 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T09:31:20.084 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.107 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.092) 0:01:31.808 ********* 2021-03-09T09:31:20.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:31:20.179 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.200 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.093) 0:01:31.901 ********* 2021-03-09T09:31:20.269 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-09T09:31:20.271 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.292 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.092) 0:01:31.993 ********* 2021-03-09T09:31:20.362 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T09:31:20.364 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.385 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.092) 0:01:32.086 ********* 2021-03-09T09:31:20.453 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:31:20.456 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.477 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.092) 0:01:32.178 ********* 2021-03-09T09:31:20.546 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-09T09:31:20.548 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.570 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.093) 0:01:32.271 ********* 2021-03-09T09:31:20.671 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-09T09:31:20.673 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.694 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.124) 0:01:32.396 ********* 2021-03-09T09:31:20.774 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-09T09:31:20.775 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.797 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.102) 0:01:32.498 ********* 2021-03-09T09:31:20.876 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-09T09:31:20.878 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:20.900 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.102) 0:01:32.601 ********* 2021-03-09T09:31:20.979 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-09T09:31:20.981 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:21.003 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:20 +0000 (0:00:00.102) 0:01:32.704 ********* 2021-03-09T09:31:21.082 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-09T09:31:21.084 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:21.110 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:21 +0000 (0:00:00.107) 0:01:32.812 ********* 2021-03-09T09:31:21.182 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-09T09:31:21.185 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:21.206 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:21 +0000 (0:00:00.095) 0:01:32.908 ********* 2021-03-09T09:31:21.275 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-09T09:31:21.277 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:21.299 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:21 +0000 (0:00:00.092) 0:01:33.000 ********* 2021-03-09T09:31:21.482 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-09T09:31:21.484 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004023", "end": "2021-03-09 09:31:21.566409", "rc": 0, "start": "2021-03-09 09:31:21.562386", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:31:21.506 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:21 +0000 (0:00:00.207) 0:01:33.207 ********* 2021-03-09T09:31:21.575 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-09T09:31:21.599 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:21 +0000 (0:00:00.093) 0:01:33.300 ********* 2021-03-09T09:31:21.982 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-09T09:31:21.984 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-09T09:31:22.007 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:22 +0000 (0:00:00.407) 0:01:33.708 ********* 2021-03-09T09:31:22.090 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-09T09:31:22.091 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:22.113 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:22 +0000 (0:00:00.105) 0:01:33.814 ********* 2021-03-09T09:31:23.234 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T09:31:23.236 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-09T09:31:24.156 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-09T09:31:24.180 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:24 +0000 (0:00:02.066) 0:01:35.881 ********* 2021-03-09T09:31:25.138 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-09T09:31:25.140 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:25.162 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:25 +0000 (0:00:00.981) 0:01:36.863 ********* 2021-03-09T09:31:25.262 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-09T09:31:25.285 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:25 +0000 (0:00:00.123) 0:01:36.986 ********* 2021-03-09T09:31:25.520 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-09T09:31:25.522 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.037272", "end": "2021-03-09 09:31:25.600161", "rc": 0, "start": "2021-03-09 09:31:25.562889", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:31:25.544 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:25 +0000 (0:00:00.258) 0:01:37.245 ********* 2021-03-09T09:31:28.397 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-09T09:31:28.399 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:28.421 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:28 +0000 (0:00:02.876) 0:01:40.122 ********* 2021-03-09T09:31:29.292 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-09T09:31:29.294 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directorie 2021-03-09T09:31:29.296 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-03-09T09:31:29.298 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T09:31:29.319 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:29 +0000 (0:00:00.897) 0:01:41.020 ********* 2021-03-09T09:31:30.245 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:31:30.247 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:31:30.275 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:30 +0000 (0:00:00.956) 0:01:41.976 ********* 2021-03-09T09:31:30.349 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-09T09:31:30.351 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:30.375 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:30 +0000 (0:00:00.099) 0:01:42.076 ********* 2021-03-09T09:31:38.806 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-09T09:31:38.808 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directorie 2021-03-09T09:31:38.810 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 15 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-03-09T09:31:38.812 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T09:31:38.833 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:38 +0000 (0:00:08.457) 0:01:50.534 ********* 2021-03-09T09:31:39.218 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:31:39.219 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 10.4 MB in 1s (9196 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-03-09T09:31:39.221 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-03-09T09:31:39.222 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]", "Fetched 10.4 MB in 1s (9196 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-03-09T09:31:39.224 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T09:31:39.244 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:39 +0000 (0:00:00.411) 0:01:50.945 ********* 2021-03-09T09:31:39.782 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:31:39.784 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:39.808 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:39 +0000 (0:00:00.563) 0:01:51.509 ********* 2021-03-09T09:31:39.877 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-09T09:31:39.879 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:39.901 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:39 +0000 (0:00:00.093) 0:01:51.602 ********* 2021-03-09T09:31:40.181 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-09T09:31:40.183 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:31:40.220 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:40 +0000 (0:00:00.318) 0:01:51.921 ********* 2021-03-09T09:31:41.185 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-09T09:31:41.187 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:31:41.208 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:41 +0000 (0:00:00.988) 0:01:52.909 ********* 2021-03-09T09:31:41.384 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T09:31:41.385 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-09T09:31:41.406 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:41 +0000 (0:00:00.197) 0:01:53.107 ********* 2021-03-09T09:31:41.475 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-09T09:31:41.476 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:41.498 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:41 +0000 (0:00:00.091) 0:01:53.199 ********* 2021-03-09T09:31:41.681 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-09T09:31:41.683 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-09T09:31:41.704 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:41 +0000 (0:00:00.206) 0:01:53.406 ********* 2021-03-09T09:31:42.101 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-09T09:31:42.103 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-09T09:31:42.129 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:42 +0000 (0:00:00.424) 0:01:53.830 ********* 2021-03-09T09:31:42.359 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-09T09:31:42.361 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T09:31:42.382 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:42 +0000 (0:00:00.253) 0:01:54.083 ********* 2021-03-09T09:31:42.566 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-09T09:31:42.568 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:31:42.597 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:42 +0000 (0:00:00.214) 0:01:54.298 ********* 2021-03-09T09:31:42.941 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-09T09:31:42.942 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:28:37 UTC", "ActiveEnterTimestampMonotonic": "18135251", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target network.target basic.target nss-lookup.target systemd-journald.socket sysinit.target systemd-tmpfiles-setup.service -.mount system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:28:37 UTC", "AssertTimestampMonotonic": "17798895", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:28:37 UTC", "ConditionTimestampMonotonic": "17798895", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-09T09:31:42.944 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1285", "ExecMainStartTimestamp": "Tue 2021-03-09 09:28:37 UTC", "ExecMainStartTimestampMonotonic": "18135239", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:28:37 UTC", "InactiveExitTimestampMonotonic": "17799582", "InvocationID": "e289062098e8468abbdb4a532edd09c3", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-09T09:31:42.945 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1285", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-09T09:31:42.947 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:28:37 UTC", "StateChangeTimestampMonotonic": "18135251", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:28:37 UTC", "WatchdogTimestampMonotonic": "18135250", "WatchdogUSec": "0"}} 2021-03-09T09:31:42.967 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:42 +0000 (0:00:00.370) 0:01:54.668 ********* 2021-03-09T09:31:43.036 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T09:31:43.038 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.059 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.091) 0:01:54.760 ********* 2021-03-09T09:31:43.138 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T09:31:43.139 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.166 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.107) 0:01:54.867 ********* 2021-03-09T09:31:43.236 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T09:31:43.237 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.259 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.092) 0:01:54.960 ********* 2021-03-09T09:31:43.329 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T09:31:43.331 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.353 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.094) 0:01:55.054 ********* 2021-03-09T09:31:43.423 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T09:31:43.425 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.446 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.093) 0:01:55.148 ********* 2021-03-09T09:31:43.525 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T09:31:43.528 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.549 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.102) 0:01:55.250 ********* 2021-03-09T09:31:43.619 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T09:31:43.621 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.645 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.096) 0:01:55.346 ********* 2021-03-09T09:31:43.729 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T09:31:43.731 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.755 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.109) 0:01:55.456 ********* 2021-03-09T09:31:43.824 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:31:43.826 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.847 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.092) 0:01:55.548 ********* 2021-03-09T09:31:43.937 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T09:31:43.939 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:43.960 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:43 +0000 (0:00:00.113) 0:01:55.661 ********* 2021-03-09T09:31:44.029 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:31:44.030 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.052 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.091) 0:01:55.753 ********* 2021-03-09T09:31:44.121 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.151 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.098) 0:01:55.852 ********* 2021-03-09T09:31:44.230 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T09:31:44.231 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.252 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.101) 0:01:55.954 ********* 2021-03-09T09:31:44.322 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.349 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.096) 0:01:56.050 ********* 2021-03-09T09:31:44.418 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:31:44.419 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.441 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.091) 0:01:56.142 ********* 2021-03-09T09:31:44.510 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:31:44.511 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.533 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.092) 0:01:56.234 ********* 2021-03-09T09:31:44.602 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T09:31:44.604 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.625 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.091) 0:01:56.326 ********* 2021-03-09T09:31:44.694 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T09:31:44.695 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.717 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.091) 0:01:56.418 ********* 2021-03-09T09:31:44.797 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T09:31:44.799 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.821 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.104) 0:01:56.522 ********* 2021-03-09T09:31:44.890 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T09:31:44.891 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:44.913 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.092) 0:01:56.614 ********* 2021-03-09T09:31:44.983 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:31:44.984 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:45.006 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:44 +0000 (0:00:00.092) 0:01:56.707 ********* 2021-03-09T09:31:45.075 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:31:45.077 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:45.100 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:45 +0000 (0:00:00.094) 0:01:56.802 ********* 2021-03-09T09:31:45.183 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T09:31:45.185 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:45.207 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:45 +0000 (0:00:00.106) 0:01:56.908 ********* 2021-03-09T09:31:45.276 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:31:45.278 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:45.299 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:45 +0000 (0:00:00.092) 0:01:57.000 ********* 2021-03-09T09:31:45.368 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:45.391 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:45 +0000 (0:00:00.091) 0:01:57.092 ********* 2021-03-09T09:31:45.470 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-09T09:31:45.472 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:45.497 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:45 +0000 (0:00:00.106) 0:01:57.198 ********* 2021-03-09T09:31:45.680 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-09T09:31:45.682 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-09T09:31:45.703 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:45 +0000 (0:00:00.206) 0:01:57.404 ********* 2021-03-09T09:31:46.084 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T09:31:46.086 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-09T09:31:46.107 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:46 +0000 (0:00:00.403) 0:01:57.808 ********* 2021-03-09T09:31:46.186 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T09:31:46.188 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:46.210 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:46 +0000 (0:00:00.102) 0:01:57.911 ********* 2021-03-09T09:31:46.630 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-09T09:31:46.632 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-09T09:31:46.654 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:46 +0000 (0:00:00.444) 0:01:58.355 ********* 2021-03-09T09:31:47.102 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-09T09:31:47.104 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-09T09:31:47.125 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:47 +0000 (0:00:00.470) 0:01:58.826 ********* 2021-03-09T09:31:47.510 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-09T09:31:47.511 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-09T09:31:47.533 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:47 +0000 (0:00:00.408) 0:01:59.235 ********* 2021-03-09T09:31:47.729 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-09T09:31:47.731 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:31:47.754 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:47 +0000 (0:00:00.220) 0:01:59.455 ********* 2021-03-09T09:31:47.823 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-09T09:31:47.824 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:47.826 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:47.851 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:47 +0000 (0:00:00.097) 0:01:59.552 ********* 2021-03-09T09:31:47.922 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:47.953 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:47 +0000 (0:00:00.101) 0:01:59.654 ********* 2021-03-09T09:31:48.047 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-09T09:31:48.048 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.072 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:48 +0000 (0:00:00.119) 0:01:59.773 ********* 2021-03-09T09:31:48.143 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-09T09:31:48.145 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.172 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:48 +0000 (0:00:00.095) 0:01:59.868 ********* 2021-03-09T09:31:48.242 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-09T09:31:48.244 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.265 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:48 +0000 (0:00:00.098) 0:01:59.966 ********* 2021-03-09T09:31:48.345 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:31:48.347 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T09:31:48.369 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:48 +0000 (0:00:00.104) 0:02:00.071 ********* 2021-03-09T09:31:48.470 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-09T09:31:48.472 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.481 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.483 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.492 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.503 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.505 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.515 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-09T09:31:48.526 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.528 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.538 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.539 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.549 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.560 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.562 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.588 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:48 +0000 (0:00:00.218) 0:02:00.289 ********* 2021-03-09T09:31:48.636 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:31:48.637 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 10.4 MB in 1s (9571 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2021-03-09T09:31:48.639 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2021-03-09T09:31:48.640 INFO:teuthology.task.ansible.out:u1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 15 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]", "Fetched 10.4 MB in 1s (9571 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2021-03-09T09:31:48.642 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-09T09:31:48.663 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:48 +0000 (0:00:00.074) 0:02:00.364 ********* 2021-03-09T09:31:48.839 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-09T09:31:48.841 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003110", "end": "2021-03-09 09:31:48.814117", "rc": 0, "start": "2021-03-09 09:31:48.811007", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-09T09:31:48.866 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:48 +0000 (0:00:00.202) 0:02:00.567 ********* 2021-03-09T09:31:48.945 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-09T09:31:48.946 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:48.971 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:48 +0000 (0:00:00.105) 0:02:00.672 ********* 2021-03-09T09:31:49.071 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-09T09:31:49.073 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:49.096 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:49 +0000 (0:00:00.124) 0:02:00.797 ********* 2021-03-09T09:31:49.175 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-09T09:31:49.177 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:49.202 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:49 +0000 (0:00:00.106) 0:02:00.903 ********* 2021-03-09T09:31:49.655 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-09T09:31:49.657 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:49.683 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:49 +0000 (0:00:00.480) 0:02:01.384 ********* 2021-03-09T09:31:50.127 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-09T09:31:50.129 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:31:50.152 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:50 +0000 (0:00:00.469) 0:02:01.854 ********* 2021-03-09T09:31:50.233 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-09T09:31:50.235 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:50.266 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:50 +0000 (0:00:00.113) 0:02:01.967 ********* 2021-03-09T09:31:50.385 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T09:31:50.387 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item={'block_used': 13643759, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451349, 'size_available': 927544725504, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451349, "block_size": 4096, "block_total": 240095108, "block_used": 13643759, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544725504, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-09T09:31:50.410 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:50 +0000 (0:00:00.144) 0:02:02.112 ********* 2021-03-09T09:31:50.596 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-09T09:31:50.598 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009731", "end": "2021-03-09 09:31:50.701793", "rc": 0, "start": "2021-03-09 09:31:50.692062", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Install packages with --no-install-recommends] **************** ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:50.620 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:50 +0000 (0:00:00.209) 0:02:02.321 ********* 2021-03-09T09:31:50.641 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:50 +0000 (0:00:00.021) 0:02:02.343 ********* 2021-03-09T09:31:50.753 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-09T09:31:50.755 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'block_used': 13643759, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451349, 'size_available': 927544725504, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451349, "block_size": 4096, "block_total": 240095108, "block_used": 13643759, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544725504, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:50.777 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:50 +0000 (0:00:00.135) 0:02:02.478 ********* 2021-03-09T09:31:50.794 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T09:31:50.796 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-09T09:31:50.820 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:50 +0000 (0:00:00.043) 0:02:02.522 ********* 2021-03-09T09:31:50.881 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-09T09:31:50.905 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:50 +0000 (0:00:00.084) 0:02:02.606 ********* 2021-03-09T09:31:50.911 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-09T09:31:50.913 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:50.940 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:50 +0000 (0:00:00.035) 0:02:02.641 ********* 2021-03-09T09:31:50.989 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:50.991 INFO:teuthology.task.ansible.out:TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:51.116 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-09T09:31:51.117 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-09T09:31:51.139 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:51 +0000 (0:00:00.198) 0:02:02.840 ********* 2021-03-09T09:31:51.513 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-09T09:31:51.514 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-09T09:31:51.536 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:51 +0000 (0:00:00.397) 0:02:03.238 ********* 2021-03-09T09:31:51.751 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-09T09:31:51.753 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T09:31:51.775 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:51 +0000 (0:00:00.238) 0:02:03.476 ********* 2021-03-09T09:31:51.950 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-09T09:31:51.952 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:31:51.974 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:51 +0000 (0:00:00.198) 0:02:03.675 ********* 2021-03-09T09:31:52.318 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-09T09:31:52.320 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:28:52 UTC", "ActiveEnterTimestampMonotonic": "22163404", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target -.mount basic.target systemd-journald.socket systemd-tmpfiles-setup.service system.slice network.target nss-lookup.target remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:28:52 UTC", "AssertTimestampMonotonic": "21639943", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:28:52 UTC", "ConditionTimestampMonotonic": "21639943", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-09T09:31:52.322 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1286", "ExecMainStartTimestamp": "Tue 2021-03-09 09:28:52 UTC", "ExecMainStartTimestampMonotonic": "22163393", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:28:52 UTC", "InactiveExitTimestampMonotonic": "21640629", "InvocationID": "3d844fdc942b4f22a4ced30292479caa", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-09T09:31:52.324 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1286", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-09T09:31:52.326 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:28:52 UTC", "StateChangeTimestampMonotonic": "22163404", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:28:52 UTC", "WatchdogTimestampMonotonic": "22163403", "WatchdogUSec": "0"}} 2021-03-09T09:31:52.346 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:52 +0000 (0:00:00.371) 0:02:04.047 ********* 2021-03-09T09:31:52.415 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T09:31:52.417 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:52.441 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:52 +0000 (0:00:00.094) 0:02:04.142 ********* 2021-03-09T09:31:52.524 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T09:31:52.526 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:52.557 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:52 +0000 (0:00:00.116) 0:02:04.258 ********* 2021-03-09T09:31:52.639 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T09:31:52.641 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:52.664 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:52 +0000 (0:00:00.106) 0:02:04.365 ********* 2021-03-09T09:31:52.743 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T09:31:52.745 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:52.769 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:52 +0000 (0:00:00.105) 0:02:04.470 ********* 2021-03-09T09:31:52.838 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T09:31:52.839 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:52.861 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:52 +0000 (0:00:00.091) 0:02:04.562 ********* 2021-03-09T09:31:52.931 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T09:31:52.933 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:52.954 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:52 +0000 (0:00:00.093) 0:02:04.656 ********* 2021-03-09T09:31:53.023 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:53.024 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.047 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.092) 0:02:04.748 ********* 2021-03-09T09:31:53.126 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T09:31:53.128 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.149 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.101) 0:02:04.850 ********* 2021-03-09T09:31:53.217 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:31:53.219 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.241 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.091) 0:02:04.942 ********* 2021-03-09T09:31:53.289 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:31:53.291 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCH5dqMScUTSbkXaBF8oEUOYrNP2iCMF7A'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.174686", "end": "2021-03-09 09:31:53.393939", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCH5dqMScUTSbkXaBF8oEUOYrNP2iCMF7A"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:31:51.219253", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:31:53.302 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.313 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.315 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.337 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.096) 0:02:05.038 ********* 2021-03-09T09:31:53.416 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T09:31:53.418 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.440 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.103) 0:02:05.141 ********* 2021-03-09T09:31:53.509 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T09:31:53.511 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.532 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.092) 0:02:05.234 ********* 2021-03-09T09:31:53.601 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T09:31:53.603 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.624 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.091) 0:02:05.325 ********* 2021-03-09T09:31:53.693 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T09:31:53.695 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.717 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.092) 0:02:05.418 ********* 2021-03-09T09:31:53.786 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:31:53.788 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.808 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.091) 0:02:05.510 ********* 2021-03-09T09:31:53.878 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:31:53.880 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.905 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.096) 0:02:05.606 ********* 2021-03-09T09:31:53.974 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T09:31:53.976 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:53.997 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:53 +0000 (0:00:00.092) 0:02:05.698 ********* 2021-03-09T09:31:54.066 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T09:31:54.068 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:54.089 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:54 +0000 (0:00:00.092) 0:02:05.790 ********* 2021-03-09T09:31:54.158 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T09:31:54.159 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:54.181 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:54 +0000 (0:00:00.091) 0:02:05.882 ********* 2021-03-09T09:31:54.280 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T09:31:54.282 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:54.304 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:54 +0000 (0:00:00.122) 0:02:06.005 ********* 2021-03-09T09:31:54.384 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:31:54.386 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:54.407 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:54 +0000 (0:00:00.103) 0:02:06.108 ********* 2021-03-09T09:31:54.508 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:31:54.510 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:54.531 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:54 +0000 (0:00:00.123) 0:02:06.232 ********* 2021-03-09T09:31:54.548 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:31:54.549 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCbbUwJmrcnUekBSmGyjevSuPBLw3aouka'], 'uuids': ['7fbba7be-2e67-4ca1-9a5c-bff4d279cc5c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.114338", "end": "2021-03-09 09:31:54.649556", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCbbUwJmrcnUekBSmGyjevSuPBLw3aouka"], "labels": [], "masters": [], "uuids": ["7fbba7be-2e67-4ca1-9a5c-bff4d279cc5c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:31:53.535218", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:31:54.550 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XM1Y', 'wwn-0x5000c50092935686'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XM1Y-part1', 'wwn-0x5000c50092935686-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092935686', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XM1Y', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XM1Y", "wwn-0x5000c50092935686"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XM1Y-part1", "wwn-0x5000c50092935686-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XM1Y", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092935686"}}, "skip_reason": 2021-03-09T09:31:54.552 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-09T09:31:54.604 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:54.627 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:54 +0000 (0:00:00.095) 0:02:06.328 ********* 2021-03-09T09:31:54.697 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T09:31:54.698 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:54.732 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:54 +0000 (0:00:00.104) 0:02:06.433 ********* 2021-03-09T09:31:54.804 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T09:31:54.806 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:54.827 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:54 +0000 (0:00:00.095) 0:02:06.528 ********* 2021-03-09T09:31:54.895 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-09T09:31:54.897 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:54.918 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:54 +0000 (0:00:00.091) 0:02:06.620 ********* 2021-03-09T09:31:55.101 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-09T09:31:55.103 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-09T09:31:55.124 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:55 +0000 (0:00:00.205) 0:02:06.825 ********* 2021-03-09T09:31:55.498 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T09:31:55.500 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-09T09:31:55.521 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:55 +0000 (0:00:00.397) 0:02:07.222 ********* 2021-03-09T09:31:55.600 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T09:31:55.602 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:55.624 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:55 +0000 (0:00:00.102) 0:02:07.325 ********* 2021-03-09T09:31:55.840 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:55.842 INFO:teuthology.task.ansible.out:TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-C3NAuR-8dfP-ogKd-jlPZ-Gs2o-6bBl-Jfq3n9', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204019S400BGN', 'nvme-nvme.8086-50484654363230343031395334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.160333", "end": "2021-03-09 09:31:55.936579", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-C3NAuR-8dfP-ogKd-jlPZ-Gs2o-6bBl-Jfq3n9", "nvme-INTEL_SSDPEDMD400G4_PHFT6204019S400BGN", "nvme-nvme.8086-50484654363230343031395334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:31:54.776246", "stderr": "", "stderr 2021-03-09T09:31:55.843 INFO:teuthology.task.ansible.out:_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:31:55.853 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:55.854 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:56.031 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-09T09:31:56.033 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-09T09:31:56.055 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:56 +0000 (0:00:00.430) 0:02:07.756 ********* 2021-03-09T09:31:56.446 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-09T09:31:56.448 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-09T09:31:56.469 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:56 +0000 (0:00:00.414) 0:02:08.171 ********* 2021-03-09T09:31:56.844 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-09T09:31:56.846 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-09T09:31:56.868 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:56 +0000 (0:00:00.398) 0:02:08.569 ********* 2021-03-09T09:31:57.051 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-09T09:31:57.053 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:31:57.076 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:57 +0000 (0:00:00.208) 0:02:08.777 ********* 2021-03-09T09:31:57.104 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:31:57.106 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCcytCQasiKmB4A6c8gcFpJe1zZVJpOInd'], 'uuids': ['bdbf7099-4f8e-466a-8bb8-b1f04fedeeb0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.130652", "end": "2021-03-09 09:31:57.208089", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCcytCQasiKmB4A6c8gcFpJe1zZVJpOInd"], "labels": [], "masters": [], "uuids": ["bdbf7099-4f8e-466a-8bb8-b1f04fedeeb0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:31:56.077437", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:31:57.116 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.128 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.150 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-09T09:31:57.151 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.172 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:57 +0000 (0:00:00.096) 0:02:08.874 ********* 2021-03-09T09:31:57.251 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-09T09:31:57.253 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.274 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:57 +0000 (0:00:00.101) 0:02:08.975 ********* 2021-03-09T09:31:57.342 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-09T09:31:57.344 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.369 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:57 +0000 (0:00:00.094) 0:02:09.070 ********* 2021-03-09T09:31:57.471 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-09T09:31:57.472 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.494 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:57 +0000 (0:00:00.125) 0:02:09.196 ********* 2021-03-09T09:31:57.563 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-09T09:31:57.565 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.587 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:57 +0000 (0:00:00.092) 0:02:09.288 ********* 2021-03-09T09:31:57.665 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:31:57.667 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T09:31:57.689 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:57 +0000 (0:00:00.102) 0:02:09.390 ********* 2021-03-09T09:31:57.779 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-09T09:31:57.782 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.790 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.792 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.802 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.813 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.815 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.826 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-09T09:31:57.828 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.838 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.849 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.851 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.861 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.863 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-09T09:31:57.894 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:57 +0000 (0:00:00.205) 0:02:09.595 ********* 2021-03-09T09:31:58.170 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-09T09:31:58.171 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003551", "end": "2021-03-09 09:31:58.144262", "rc": 0, "start": "2021-03-09 09:31:58.140711", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-09T09:31:58.200 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:58 +0000 (0:00:00.305) 0:02:09.901 ********* 2021-03-09T09:31:58.280 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-09T09:31:58.282 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:58.307 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:58 +0000 (0:00:00.106) 0:02:10.008 ********* 2021-03-09T09:31:58.377 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:31:58.379 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXC3wBWEzEb5PeygV8lcyEk3je3vheI292Z'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.131319", "end": "2021-03-09 09:31:58.483460", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXC3wBWEzEb5PeygV8lcyEk3je3vheI292Z"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:31:57.352141", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:31:58.389 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:58.390 INFO:teuthology.task.ansible.out:TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:31:58.391 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:58.414 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:58 +0000 (0:00:00.106) 0:02:10.115 ********* 2021-03-09T09:31:58.493 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:58.517 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:58 +0000 (0:00:00.102) 0:02:10.218 ********* 2021-03-09T09:31:59.475 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-09T09:31:59.476 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:31:59.501 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:59 +0000 (0:00:00.984) 0:02:11.203 ********* 2021-03-09T09:31:59.571 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-09T09:31:59.573 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:59.595 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:59 +0000 (0:00:00.093) 0:02:11.296 ********* 2021-03-09T09:31:59.623 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:31:59.625 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCdmhwkE5TodsItfai0ni3Y4dND2TeQVcn'], 'uuids': ['b1cac119-ad8d-46b0-826a-99e39c0bbc84']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.115255", "end": "2021-03-09 09:31:59.733086", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCdmhwkE5TodsItfai0ni3Y4dND2TeQVcn"], "labels": [], "masters": [], "uuids": ["b1cac119-ad8d-46b0-826a-99e39c0bbc84"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:31:58.617831", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:31:59.626 INFO:teuthology.task.ansible.out: 2021-03-09T09:31:59.661 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:59 +0000 (0:00:00.065) 0:02:11.362 ********* 2021-03-09T09:31:59.677 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T09:31:59.679 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-09T09:31:59.704 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:59 +0000 (0:00:00.042) 0:02:11.405 ********* 2021-03-09T09:31:59.741 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T09:31:59.743 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:31:59.888 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-09T09:31:59.890 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.008755", "end": "2021-03-09 09:31:59.973267", "rc": 0, "start": "2021-03-09 09:31:59.964512", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:31:59.912 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:31:59 +0000 (0:00:00.208) 0:02:11.613 ********* 2021-03-09T09:31:59.991 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-09T09:31:59.993 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:00.015 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:00 +0000 (0:00:00.103) 0:02:11.717 ********* 2021-03-09T09:32:00.033 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T09:32:00.034 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCH5dqMScUTSbkXaBF8oEUOYrNP2iCMF7A'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.156974", "end": "2021-03-09 09:32:00.130653", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCH5dqMScUTSbkXaBF8oEUOYrNP2iCMF7A"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:31:59.973679", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152431 s, 757 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152431 s, 757 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:00.036 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:00.047 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:00.089 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-09T09:32:00.112 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:00 +0000 (0:00:00.097) 0:02:11.814 ********* 2021-03-09T09:32:00.201 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:32:00.203 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:00.328 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T09:32:00.330 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCbbUwJmrcnUekBSmGyjevSuPBLw3aouka'], 'uuids': ['7fbba7be-2e67-4ca1-9a5c-bff4d279cc5c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.153526", "end": "2021-03-09 09:32:00.441007", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCbbUwJmrcnUekBSmGyjevSuPBLw3aouka"], "labels": [], "masters": [], "uuids": ["7fbba7be-2e67-4ca1-9a5c-bff4d279cc5c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:00.287481", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148804 s, 775 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148804 s, 775 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:00.340 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XM1Y', 'wwn-0x5000c50092935686'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XM1Y-part1', 'wwn-0x5000c50092935686-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092935686', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XM1Y', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XM1Y", "wwn-0x5000c50092935686"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XM1Y-part1", "wwn-0x5000c50092935686-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XM1Y", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092935686"}}, "skip_reason": 2021-03-09T09:32:00.342 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-09T09:32:00.508 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-C3NAuR-8dfP-ogKd-jlPZ-Gs2o-6bBl-Jfq3n9', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204019S400BGN', 'nvme-nvme.8086-50484654363230343031395334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.043761", "end": "2021-03-09 09:32:00.608581", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-C3NAuR-8dfP-ogKd-jlPZ-Gs2o-6bBl-Jfq3n9", "nvme-INTEL_SSDPEDMD400G4_PHFT6204019S400BGN", "nvme-nvme.8086-50484654363230343031395334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:00.564820", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0394339 s, 2.9 GB/s", "std 2021-03-09T09:32:00.510 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0394339 s, 2.9 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:00.519 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:00.521 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:00.794 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCcytCQasiKmB4A6c8gcFpJe1zZVJpOInd'], 'uuids': ['bdbf7099-4f8e-466a-8bb8-b1f04fedeeb0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.147191", "end": "2021-03-09 09:32:00.897812", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCcytCQasiKmB4A6c8gcFpJe1zZVJpOInd"], "labels": [], "masters": [], "uuids": ["bdbf7099-4f8e-466a-8bb8-b1f04fedeeb0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:00.750621", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14412 s, 800 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14412 s, 800 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:00.806 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:00.807 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:01.100 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXC3wBWEzEb5PeygV8lcyEk3je3vheI292Z'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.159347", "end": "2021-03-09 09:32:01.196714", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXC3wBWEzEb5PeygV8lcyEk3je3vheI292Z"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:01.037367", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154545 s, 746 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154545 s, 746 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:01.101 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:01.392 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCdmhwkE5TodsItfai0ni3Y4dND2TeQVcn'], 'uuids': ['b1cac119-ad8d-46b0-826a-99e39c0bbc84']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.154738", "end": "2021-03-09 09:32:01.489939", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-7Jfm1jEZtzqQCTOKNdSHpNpJuIOsvLXCdmhwkE5TodsItfai0ni3Y4dND2TeQVcn"], "labels": [], "masters": [], "uuids": ["b1cac119-ad8d-46b0-826a-99e39c0bbc84"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:01.335201", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149963 s, 769 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149963 s, 769 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:01.427 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:01 +0000 (0:00:01.314) 0:02:13.128 ********* 2021-03-09T09:32:01.735 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-09T09:32:01.737 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.123768", "end": "2021-03-09 09:32:01.792450", "rc": 0, "start": "2021-03-09 09:32:01.668682", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:01.759 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:01 +0000 (0:00:00.332) 0:02:13.460 ********* 2021-03-09T09:32:01.964 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-09T09:32:01.966 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.068443", "end": "2021-03-09 09:32:02.069419", "rc": 0, "start": "2021-03-09 09:32:02.000976", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:01.987 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:01 +0000 (0:00:00.228) 0:02:13.689 ********* 2021-03-09T09:32:02.056 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-09T09:32:02.120 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.132) 0:02:13.821 ********* 2021-03-09T09:32:02.148 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-09T09:32:02.150 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.173 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.053) 0:02:13.875 ********* 2021-03-09T09:32:02.201 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-09T09:32:02.202 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.223 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.049) 0:02:13.925 ********* 2021-03-09T09:32:02.250 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T09:32:02.252 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.275 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.051) 0:02:13.976 ********* 2021-03-09T09:32:02.307 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-09T09:32:02.309 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.331 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.055) 0:02:14.032 ********* 2021-03-09T09:32:02.369 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:32:02.371 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYb7MXgb1uJwBti31qAdVBJ2mI3Du1it8G'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.044806", "end": "2021-03-09 09:32:02.450788", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYb7MXgb1uJwBti31qAdVBJ2mI3Du1it8G"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:00.405982", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:32:02.381 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.382 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.467 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-09T09:32:02.469 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:32:02.491 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.159) 0:02:14.192 ********* 2021-03-09T09:32:02.529 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T09:32:02.530 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'block_used': 13643759, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451349, 'size_available': 927544725504, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451349, "block_size": 4096, "block_total": 240095108, "block_used": 13643759, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544725504, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.553 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.062) 0:02:14.255 ********* 2021-03-09T09:32:02.580 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-09T09:32:02.582 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.602 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.048) 0:02:14.303 ********* 2021-03-09T09:32:02.629 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:02.631 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.652 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.049) 0:02:14.353 ********* 2021-03-09T09:32:02.678 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:02.680 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.705 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.052) 0:02:14.406 ********* 2021-03-09T09:32:02.733 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-09T09:32:02.735 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.756 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.051) 0:02:14.457 ********* 2021-03-09T09:32:02.783 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-09T09:32:02.785 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.806 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.050) 0:02:14.508 ********* 2021-03-09T09:32:02.833 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-09T09:32:02.835 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:02.857 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:02 +0000 (0:00:00.049) 0:02:14.558 ********* 2021-03-09T09:32:03.483 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-09T09:32:03.486 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T09:32:03.507 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:03 +0000 (0:00:00.650) 0:02:15.208 ********* 2021-03-09T09:32:03.650 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:32:03.652 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYBVeZYukoqitCWeZlEJ9M2Od9ukB7xwm6'], 'uuids': ['4726d5fa-4168-4572-b0be-4eb6b9b322f8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.133722", "end": "2021-03-09 09:32:03.724047", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYBVeZYukoqitCWeZlEJ9M2Od9ukB7xwm6"], "labels": [], "masters": [], "uuids": ["4726d5fa-4168-4572-b0be-4eb6b9b322f8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:02.590325", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi143.front.sepia.ceph.com] => 2021-03-09T09:32:03.654 INFO:teuthology.task.ansible.out:(item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFM8', 'wwn-0x5000c50092923e43'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFM8-part1', 'wwn-0x5000c50092923e43-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092923e43', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFM8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFM8", "wwn-0x5000c50092923e43"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFM8-part1", "wwn-0x5000c50092923e43-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFM8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092923e43"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:04.102 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T09:32:04.104 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:04.384 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:04.655 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:04.895 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:05.000 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:32:05.002 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-e6ml2B-NX9m-HjYS-D6sd-OhnQ-cFTz-7c1Fud', 'nvme-INTEL_SSDPEDMD400G4_CVFT62330073400BGN', 'nvme-nvme.8086-43564654363233333030373334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.213106", "end": "2021-03-09 09:32:05.076676", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-e6ml2B-NX9m-HjYS-D6sd-OhnQ-cFTz-7c1Fud", "nvme-INTEL_SSDPEDMD400G4_CVFT62330073400BGN", "nvme-nvme.8086-43564654363233333030373334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:03.863570", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-09T09:32:05.003 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:05.014 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:05.139 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T09:32:05.141 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:05.165 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:05 +0000 (0:00:01.658) 0:02:16.867 ********* 2021-03-09T09:32:05.307 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T09:32:05.308 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T09:32:05.330 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:05 +0000 (0:00:00.164) 0:02:17.031 ********* 2021-03-09T09:32:05.482 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T09:32:05.484 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:32:05.598 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:32:05.713 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:32:05.839 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:32:05.850 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:05.876 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:05 +0000 (0:00:00.545) 0:02:17.577 ********* 2021-03-09T09:32:05.903 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-09T09:32:05.905 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:05.926 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:05 +0000 (0:00:00.050) 0:02:17.628 ********* 2021-03-09T09:32:05.954 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-09T09:32:05.956 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:05.977 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:05 +0000 (0:00:00.050) 0:02:17.678 ********* 2021-03-09T09:32:06.119 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-09T09:32:06.121 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:32:06.142 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:06 +0000 (0:00:00.164) 0:02:17.843 ********* 2021-03-09T09:32:06.180 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-09T09:32:06.181 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T09:32:06.208 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:06 +0000 (0:00:00.065) 0:02:17.909 ********* 2021-03-09T09:32:06.266 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:32:06.268 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYKxWo13g6fCi5xgyZV67ih1oXZh2BAspY'], 'uuids': ['d1af078c-6169-43fc-8124-063c3ee69e66']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.131355", "end": "2021-03-09 09:32:06.350571", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYKxWo13g6fCi5xgyZV67ih1oXZh2BAspY"], "labels": [], "masters": [], "uuids": ["d1af078c-6169-43fc-8124-063c3ee69e66"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:05.219216", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:32:06.279 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:06.290 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:06.719 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi173.front.sepia.ceph.com] => {"changed": true} 2021-03-09T09:32:06.744 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:06 +0000 (0:00:00.536) 0:02:18.445 ********* 2021-03-09T09:32:07.185 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-09T09:32:07.187 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T09:32:07.210 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:07 +0000 (0:00:00.466) 0:02:18.912 ********* 2021-03-09T09:32:07.237 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-09T09:32:07.239 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:07.260 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:07 +0000 (0:00:00.049) 0:02:18.961 ********* 2021-03-09T09:32:07.286 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-09T09:32:07.288 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:07.308 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:07 +0000 (0:00:00.048) 0:02:19.010 ********* 2021-03-09T09:32:07.334 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:07.357 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:07 +0000 (0:00:00.049) 0:02:19.059 ********* 2021-03-09T09:32:07.384 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T09:32:07.386 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:07.406 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:07 +0000 (0:00:00.048) 0:02:19.107 ********* 2021-03-09T09:32:07.546 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-09T09:32:07.548 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T09:32:07.570 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:07 +0000 (0:00:00.164) 0:02:19.271 ********* 2021-03-09T09:32:07.576 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:32:07.578 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYE7571t1mp8PsyFIK6xzRY0eOP6ECdP8Z'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.142676", "end": "2021-03-09 09:32:07.633845", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYE7571t1mp8PsyFIK6xzRY0eOP6ECdP8Z"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:06.491169", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:32:07.580 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:07.708 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-09T09:32:07.709 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:32:07.731 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:07 +0000 (0:00:00.161) 0:02:19.432 ********* 2021-03-09T09:32:08.079 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-09T09:32:08.081 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-09T09:32:08.102 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:08 +0000 (0:00:00.370) 0:02:19.803 ********* 2021-03-09T09:32:08.434 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-09T09:32:08.435 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-09T09:32:08.457 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:08 +0000 (0:00:00.354) 0:02:20.158 ********* 2021-03-09T09:32:08.811 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T09:32:08.812 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYve9dB4noI3jlwp39TNpIfvuYqfgWrzLO'], 'uuids': ['2a2fd1b0-afc8-4408-a390-0408a3948042']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.126960", "end": "2021-03-09 09:32:08.892580", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYve9dB4noI3jlwp39TNpIfvuYqfgWrzLO"], "labels": [], "masters": [], "uuids": ["2a2fd1b0-afc8-4408-a390-0408a3948042"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:07.765620", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T09:32:08.855 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:08 +0000 (0:00:00.398) 0:02:20.557 ********* 2021-03-09T09:32:08.904 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T09:32:08.906 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:09.196 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYb7MXgb1uJwBti31qAdVBJ2mI3Du1it8G'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.156183", "end": "2021-03-09 09:32:09.269732", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYb7MXgb1uJwBti31qAdVBJ2mI3Du1it8G"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:09.113549", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153031 s, 754 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153031 s, 754 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:09.207 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:09.208 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:09.364 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-09T09:32:09.365 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:32:09.388 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:09 +0000 (0:00:00.532) 0:02:21.089 ********* 2021-03-09T09:32:09.498 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T09:32:09.500 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYBVeZYukoqitCWeZlEJ9M2Od9ukB7xwm6'], 'uuids': ['4726d5fa-4168-4572-b0be-4eb6b9b322f8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.162312", "end": "2021-03-09 09:32:09.576606", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYBVeZYukoqitCWeZlEJ9M2Od9ukB7xwm6"], "labels": [], "masters": [], "uuids": ["4726d5fa-4168-4572-b0be-4eb6b9b322f8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:09.414294", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.157552 s, 732 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.157552 s, 732 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:09.510 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFM8', 'wwn-0x5000c50092923e43'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XFM8-part1', 'wwn-0x5000c50092923e43-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092923e43', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XFM8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFM8", "wwn-0x5000c50092923e43"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XFM8-part1", "wwn-0x5000c50092923e43-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XFM8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092923e43"}}, "skip_reason": 2021-03-09T09:32:09.512 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-09T09:32:09.585 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-09T09:32:09.586 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047633", "end": "2021-03-09 09:32:09.689842", "rc": 0, "start": "2021-03-09 09:32:09.642209", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-09T09:32:09.608 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:09 +0000 (0:00:00.220) 0:02:21.309 ********* 2021-03-09T09:32:09.646 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-09T09:32:09.647 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:09.670 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:09 +0000 (0:00:00.062) 0:02:21.372 ********* 2021-03-09T09:32:09.688 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T09:32:09.689 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-e6ml2B-NX9m-HjYS-D6sd-OhnQ-cFTz-7c1Fud', 'nvme-INTEL_SSDPEDMD400G4_CVFT62330073400BGN', 'nvme-nvme.8086-43564654363233333030373334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052534", "end": "2021-03-09 09:32:09.761784", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-e6ml2B-NX9m-HjYS-D6sd-OhnQ-cFTz-7c1Fud", "nvme-INTEL_SSDPEDMD400G4_CVFT62330073400BGN", "nvme-nvme.8086-43564654363233333030373334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:09.709250", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0478288 s, 2.4 GB/s", "std 2021-03-09T09:32:09.691 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0478288 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:09.701 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:09.702 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:09.702 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:09.726 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:09 +0000 (0:00:00.055) 0:02:21.427 ********* 2021-03-09T09:32:09.982 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYKxWo13g6fCi5xgyZV67ih1oXZh2BAspY'], 'uuids': ['d1af078c-6169-43fc-8124-063c3ee69e66']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.146596", "end": "2021-03-09 09:32:10.054951", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYKxWo13g6fCi5xgyZV67ih1oXZh2BAspY"], "labels": [], "masters": [], "uuids": ["d1af078c-6169-43fc-8124-063c3ee69e66"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:09.908355", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144288 s, 799 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144288 s, 799 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:09.993 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:09.995 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:10.285 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYE7571t1mp8PsyFIK6xzRY0eOP6ECdP8Z'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.159454", "end": "2021-03-09 09:32:10.360606", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYE7571t1mp8PsyFIK6xzRY0eOP6ECdP8Z"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:10.201152", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154616 s, 746 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154616 s, 746 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:10.297 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:10.575 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYve9dB4noI3jlwp39TNpIfvuYqfgWrzLO'], 'uuids': ['2a2fd1b0-afc8-4408-a390-0408a3948042']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.156984", "end": "2021-03-09 09:32:10.651483", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-tWVZTDaJocGmDtEEuMndWSRXXOIr8sXYve9dB4noI3jlwp39TNpIfvuYqfgWrzLO"], "labels": [], "masters": [], "uuids": ["2a2fd1b0-afc8-4408-a390-0408a3948042"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 09:32:10.494499", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152594 s, 756 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152594 s, 756 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:10.613 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:10 +0000 (0:00:00.886) 0:02:22.314 ********* 2021-03-09T09:32:10.634 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-09T09:32:10.636 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:32:10.661 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:10 +0000 (0:00:00.048) 0:02:22.363 ********* 2021-03-09T09:32:10.689 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-09T09:32:10.691 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:10.713 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:10 +0000 (0:00:00.051) 0:02:22.414 ********* 2021-03-09T09:32:10.948 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-09T09:32:10.950 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.188913", "end": "2021-03-09 09:32:11.020603", "rc": 0, "start": "2021-03-09 09:32:10.831690", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:10.973 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:10 +0000 (0:00:00.260) 0:02:22.674 ********* 2021-03-09T09:32:11.042 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-09T09:32:11.044 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-09T09:32:11.068 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.094) 0:02:22.769 ********* 2021-03-09T09:32:11.095 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-09T09:32:11.097 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.118 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.050) 0:02:22.820 ********* 2021-03-09T09:32:11.167 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-09T09:32:11.169 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.046215", "end": "2021-03-09 09:32:11.249720", "rc": 0, "start": "2021-03-09 09:32:11.203505", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:11.192 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.073) 0:02:22.893 ********* 2021-03-09T09:32:11.221 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-09T09:32:11.247 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.054) 0:02:22.948 ********* 2021-03-09T09:32:11.275 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-09T09:32:11.277 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.303 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.056) 0:02:23.005 ********* 2021-03-09T09:32:11.330 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-09T09:32:11.332 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-09T09:32:11.437 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:28:37 UTC", "ActiveEnterTimestampMonotonic": "17313013", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service -.mount systemd-journald.socket sysinit.target basic.target network.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:28:36 UTC", "AssertTimestampMonotonic": "17152141", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:28:36 UTC", "ConditionTimestampMonotonic": "17152141", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-09T09:32:11.439 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1004", "ExecMainStartTimestamp": "Tue 2021-03-09 09:28:37 UTC", "ExecMainStartTimestampMonotonic": "17312991", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:28:36 UTC", "InactiveExitTimestampMonotonic": "17152965", "InvocationID": "b4983a5e11ec418796453a4a289cab92", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-09T09:32:11.440 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1004", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:28:37 UTC", "S 2021-03-09T09:32:11.442 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "17313013", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:28:37 UTC", "WatchdogTimestampMonotonic": "17313012", "WatchdogUSec": "0"}} 2021-03-09T09:32:11.459 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.156) 0:02:23.161 ********* 2021-03-09T09:32:11.481 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.021) 0:02:23.182 ********* 2021-03-09T09:32:11.487 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T09:32:11.489 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.514 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.033) 0:02:23.216 ********* 2021-03-09T09:32:11.521 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:11.522 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T09:32:11.546 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.032) 0:02:23.248 ********* 2021-03-09T09:32:11.553 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-09T09:32:11.554 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.578 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.031) 0:02:23.279 ********* 2021-03-09T09:32:11.584 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:11.585 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.611 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.032) 0:02:23.312 ********* 2021-03-09T09:32:11.638 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-09T09:32:11.640 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.662 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.051) 0:02:23.363 ********* 2021-03-09T09:32:11.710 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-09T09:32:11.712 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T09:32:11.735 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.072) 0:02:23.436 ********* 2021-03-09T09:32:11.773 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T09:32:11.775 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.798 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.063) 0:02:23.499 ********* 2021-03-09T09:32:11.825 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-09T09:32:11.827 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.849 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.050) 0:02:23.550 ********* 2021-03-09T09:32:11.875 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:11.877 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.900 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.050) 0:02:23.601 ********* 2021-03-09T09:32:11.927 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:11.929 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:11.956 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:11 +0000 (0:00:00.056) 0:02:23.657 ********* 2021-03-09T09:32:11.984 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-09T09:32:11.985 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:12.007 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:12 +0000 (0:00:00.050) 0:02:23.708 ********* 2021-03-09T09:32:12.034 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-09T09:32:12.036 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:12.057 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:12 +0000 (0:00:00.050) 0:02:23.759 ********* 2021-03-09T09:32:12.084 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-09T09:32:12.086 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:12.107 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:12 +0000 (0:00:00.049) 0:02:23.809 ********* 2021-03-09T09:32:12.456 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-09T09:32:12.458 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T09:32:12.480 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:12 +0000 (0:00:00.372) 0:02:24.181 ********* 2021-03-09T09:32:12.529 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-09T09:32:12.530 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:32:12.554 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:12 +0000 (0:00:00.073) 0:02:24.255 ********* 2021-03-09T09:32:12.582 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-09T09:32:12.584 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:12.605 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:12 +0000 (0:00:00.051) 0:02:24.306 ********* 2021-03-09T09:32:12.746 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-09T09:32:12.748 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T09:32:12.770 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:12 +0000 (0:00:00.164) 0:02:24.471 ********* 2021-03-09T09:32:12.799 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T09:32:12.801 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:13.060 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:13.091 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-09T09:32:13.093 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-09T09:32:13.115 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:13 +0000 (0:00:00.344) 0:02:24.816 ********* 2021-03-09T09:32:13.308 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T09:32:13.310 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:13.538 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:13.784 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T09:32:13.809 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:13 +0000 (0:00:00.693) 0:02:25.510 ********* 2021-03-09T09:32:13.941 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T09:32:13.942 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T09:32:13.964 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:13 +0000 (0:00:00.155) 0:02:25.665 ********* 2021-03-09T09:32:14.043 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-09T09:32:14.045 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-09T09:32:14.068 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:14 +0000 (0:00:00.103) 0:02:25.769 ********* 2021-03-09T09:32:14.116 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T09:32:14.117 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:32:14.232 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:32:14.349 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:32:14.463 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T09:32:14.474 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:14.507 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:14 +0000 (0:00:00.439) 0:02:26.208 ********* 2021-03-09T09:32:14.536 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-09T09:32:14.538 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:14.560 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:14 +0000 (0:00:00.052) 0:02:26.261 ********* 2021-03-09T09:32:14.587 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-09T09:32:14.588 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:14.610 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:14 +0000 (0:00:00.050) 0:02:26.312 ********* 2021-03-09T09:32:14.617 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-09T09:32:14.618 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false} 2021-03-09T09:32:14.644 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:14 +0000 (0:00:00.033) 0:02:26.345 ********* 2021-03-09T09:32:14.671 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-09T09:32:14.673 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-09T09:32:14.696 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:14 +0000 (0:00:00.052) 0:02:26.397 ********* 2021-03-09T09:32:14.745 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-09T09:32:14.746 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:32:14.779 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:14 +0000 (0:00:00.082) 0:02:26.480 ********* 2021-03-09T09:32:14.820 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-09T09:32:14.822 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T09:32:14.859 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:14 +0000 (0:00:00.080) 0:02:26.560 ********* 2021-03-09T09:32:14.870 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-09T09:32:14.871 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615282138.127695, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615282138.1236951, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615282138.1236951, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-09T09:32:14.907 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:14 +0000 (0:00:00.048) 0:02:26.608 ********* 2021-03-09T09:32:15.091 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-03-09T09:32:15.092 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true} 2021-03-09T09:32:15.114 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.207) 0:02:26.816 ********* 2021-03-09T09:32:15.163 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-09T09:32:15.165 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-09T09:32:15.188 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.073) 0:02:26.889 ********* 2021-03-09T09:32:15.214 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-09T09:32:15.216 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T09:32:15.237 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.049) 0:02:26.938 ********* 2021-03-09T09:32:15.264 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.286 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.048) 0:02:26.987 ********* 2021-03-09T09:32:15.312 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-09T09:32:15.314 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T09:32:15.337 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.051) 0:02:27.038 ********* 2021-03-09T09:32:15.364 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-09T09:32:15.365 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.386 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.049) 0:02:27.088 ********* 2021-03-09T09:32:15.413 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-09T09:32:15.415 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.436 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.049) 0:02:27.137 ********* 2021-03-09T09:32:15.456 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.020) 0:02:27.157 ********* 2021-03-09T09:32:15.461 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T09:32:15.463 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.488 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.031) 0:02:27.189 ********* 2021-03-09T09:32:15.515 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T09:32:15.517 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.538 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.050) 0:02:27.239 ********* 2021-03-09T09:32:15.585 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-09T09:32:15.587 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T09:32:15.618 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.080) 0:02:27.319 ********* 2021-03-09T09:32:15.647 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-09T09:32:15.649 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.671 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.052) 0:02:27.372 ********* 2021-03-09T09:32:15.687 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-09T09:32:15.689 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T09:32:15.714 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.042) 0:02:27.415 ********* 2021-03-09T09:32:15.721 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-09T09:32:15.723 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi173.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.723 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.723 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.750 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.036) 0:02:27.451 ********* 2021-03-09T09:32:15.789 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-09T09:32:15.790 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.800 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.801 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.802 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.812 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.838 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.087) 0:02:27.539 ********* 2021-03-09T09:32:15.865 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:15.867 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Add CPAN config directory for the root user.] ***************** ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:32:15.889 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.051) 0:02:27.590 ********* 2021-03-09T09:32:15.913 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.023) 0:02:27.614 ********* 2021-03-09T09:32:15.941 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-09T09:32:15.943 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:15.966 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:15 +0000 (0:00:00.052) 0:02:27.667 ********* 2021-03-09T09:32:15.992 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-09T09:32:15.994 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:16.019 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.052) 0:02:27.720 ********* 2021-03-09T09:32:16.046 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-09T09:32:16.048 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:16.070 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.051) 0:02:27.771 ********* 2021-03-09T09:32:16.097 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-09T09:32:16.099 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:16.122 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.052) 0:02:27.823 ********* 2021-03-09T09:32:16.149 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T09:32:16.151 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:16.174 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.051) 0:02:27.875 ********* 2021-03-09T09:32:16.200 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-09T09:32:16.202 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:16.234 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.060) 0:02:27.936 ********* 2021-03-09T09:32:16.241 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-09T09:32:16.242 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-09T09:32:16.266 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.032) 0:02:27.968 ********* 2021-03-09T09:32:16.273 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T09:32:16.275 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:16.299 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.032) 0:02:28.000 ********* 2021-03-09T09:32:16.462 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-09T09:32:16.464 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011048", "end": "2021-03-09 09:32:16.427492", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 09:32:16.416444", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:16.578 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006109", "end": "2021-03-09 09:32:16.553045", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 09:32:16.546936", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:16.600 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-09T09:32:16.602 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-09T09:32:16.626 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.326) 0:02:28.327 ********* 2021-03-09T09:32:16.696 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-09T09:32:16.698 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005524", "end": "2021-03-09 09:32:16.672800", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 09:32:16.667276", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:16.822 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.017169", "end": "2021-03-09 09:32:16.798543", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 09:32:16.781374", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:16.848 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.222) 0:02:28.550 ********* 2021-03-09T09:32:16.875 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-09T09:32:16.898 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:16 +0000 (0:00:00.050) 0:02:28.600 ********* 2021-03-09T09:32:17.029 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-09T09:32:17.031 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-09T09:32:17.551 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-09T09:32:17.553 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:32:17.575 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:17 +0000 (0:00:00.676) 0:02:29.276 ********* 2021-03-09T09:32:17.758 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-09T09:32:17.760 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.045738", "end": "2021-03-09 09:32:17.841164", "rc": 0, "start": "2021-03-09 09:32:17.795426", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-09T09:32:17.781 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:17 +0000 (0:00:00.206) 0:02:29.483 ********* 2021-03-09T09:32:17.808 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-09T09:32:17.810 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:17.837 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:17 +0000 (0:00:00.055) 0:02:29.538 ********* 2021-03-09T09:32:17.864 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-09T09:32:17.866 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:17.887 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:17 +0000 (0:00:00.050) 0:02:29.588 ********* 2021-03-09T09:32:18.763 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-09T09:32:18.764 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:32:18.786 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:18 +0000 (0:00:00.899) 0:02:30.487 ********* 2021-03-09T09:32:18.813 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:18.836 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:18 +0000 (0:00:00.049) 0:02:30.537 ********* 2021-03-09T09:32:19.164 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-09T09:32:19.165 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-09T09:32:19.187 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:19 +0000 (0:00:00.350) 0:02:30.888 ********* 2021-03-09T09:32:19.214 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-09T09:32:19.215 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:19.238 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:19 +0000 (0:00:00.051) 0:02:30.939 ********* 2021-03-09T09:32:19.441 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-09T09:32:19.444 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 09:28:51 UTC", "ActiveEnterTimestampMonotonic": "21236430", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket -.mount network.target basic.target systemd-tmpfiles-setup.service sysinit.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 09:28:51 UTC", "AssertTimestampMonotonic": "21064140", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 09:28:51 UTC", "ConditionTimestampMonotonic": "21064139", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-09T09:32:19.445 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1047", "ExecMainStartTimestamp": "Tue 2021-03-09 09:28:51 UTC", "ExecMainStartTimestampMonotonic": "21236409", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 09:28:51 UTC", "InactiveExitTimestampMonotonic": "21064932", "InvocationID": "48865ce6430f429dbd746935b1956759", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-09T09:32:19.447 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1047", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:28:51 UTC", "S 2021-03-09T09:32:19.449 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "21236430", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 09:28:51 UTC", "WatchdogTimestampMonotonic": "21236429", "WatchdogUSec": "0"}} 2021-03-09T09:32:19.466 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:19 +0000 (0:00:00.228) 0:02:31.167 ********* 2021-03-09T09:32:19.493 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:19.495 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T09:32:19.517 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:19 +0000 (0:00:00.050) 0:02:31.218 ********* 2021-03-09T09:32:19.543 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:19.545 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:19.568 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:19 +0000 (0:00:00.051) 0:02:31.269 ********* 2021-03-09T09:32:19.595 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-09T09:32:19.597 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:19.618 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:19 +0000 (0:00:00.050) 0:02:31.320 ********* 2021-03-09T09:32:20.468 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-09T09:32:20.470 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:32:20.491 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:20 +0000 (0:00:00.872) 0:02:32.193 ********* 2021-03-09T09:32:20.518 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-09T09:32:20.520 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:20.542 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:20 +0000 (0:00:00.050) 0:02:32.243 ********* 2021-03-09T09:32:20.683 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-09T09:32:20.685 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T09:32:20.707 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:20 +0000 (0:00:00.164) 0:02:32.408 ********* 2021-03-09T09:32:21.117 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-09T09:32:21.119 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-09T09:32:21.141 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:21 +0000 (0:00:00.433) 0:02:32.842 ********* 2021-03-09T09:32:21.934 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-09T09:32:21.936 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-09T09:32:21.958 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:21 +0000 (0:00:00.817) 0:02:33.659 ********* 2021-03-09T09:32:22.472 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-09T09:32:22.473 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false} 2021-03-09T09:32:22.494 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:22 +0000 (0:00:00.536) 0:02:34.196 ********* 2021-03-09T09:32:22.522 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-09T09:32:22.524 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-09T09:32:22.544 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:22 +0000 (0:00:00.049) 0:02:34.246 ********* 2021-03-09T09:32:22.695 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-09T09:32:22.697 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615282152.9998004, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615282152.9958005, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615282152.9958005, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-09T09:32:22.719 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:22 +0000 (0:00:00.174) 0:02:34.420 ********* 2021-03-09T09:32:22.849 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-09T09:32:22.851 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-09T09:32:22.872 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:22 +0000 (0:00:00.152) 0:02:34.573 ********* 2021-03-09T09:32:22.898 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-09T09:32:22.900 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T09:32:22.920 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:22 +0000 (0:00:00.048) 0:02:34.621 ********* 2021-03-09T09:32:22.947 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-09T09:32:22.949 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:22.969 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:22 +0000 (0:00:00.049) 0:02:34.671 ********* 2021-03-09T09:32:23.099 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-09T09:32:23.100 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-09T09:32:23.122 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.152) 0:02:34.823 ********* 2021-03-09T09:32:23.251 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-09T09:32:23.253 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T09:32:23.275 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.153) 0:02:34.976 ********* 2021-03-09T09:32:23.302 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-09T09:32:23.304 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.324 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.049) 0:02:35.025 ********* 2021-03-09T09:32:23.362 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-09T09:32:23.364 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.374 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.375 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.398 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.073) 0:02:35.099 ********* 2021-03-09T09:32:23.435 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-09T09:32:23.437 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi143.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.447 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.448 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.449 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.482 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.083) 0:02:35.183 ********* 2021-03-09T09:32:23.508 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T09:32:23.510 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.536 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.054) 0:02:35.237 ********* 2021-03-09T09:32:23.562 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-09T09:32:23.564 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.585 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.048) 0:02:35.286 ********* 2021-03-09T09:32:23.610 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-09T09:32:23.612 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.634 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.049) 0:02:35.336 ********* 2021-03-09T09:32:23.660 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-09T09:32:23.662 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.682 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.048) 0:02:35.384 ********* 2021-03-09T09:32:23.710 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-09T09:32:23.712 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.734 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.051) 0:02:35.435 ********* 2021-03-09T09:32:23.771 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T09:32:23.772 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.795 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.060) 0:02:35.496 ********* 2021-03-09T09:32:23.821 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-09T09:32:23.823 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.846 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.050) 0:02:35.547 ********* 2021-03-09T09:32:23.872 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T09:32:23.874 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:23.894 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:23 +0000 (0:00:00.048) 0:02:35.596 ********* 2021-03-09T09:32:24.065 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-09T09:32:24.067 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011023", "end": "2021-03-09 09:32:24.143552", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 09:32:24.132529", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:24.191 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005913", "end": "2021-03-09 09:32:24.268372", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 09:32:24.262459", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:24.305 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.004775", "end": "2021-03-09 09:32:24.382605", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 09:32:24.377830", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:24.430 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.007678", "end": "2021-03-09 09:32:24.509507", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 09:32:24.501829", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:24.460 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:24 +0000 (0:00:00.565) 0:02:36.161 ********* 2021-03-09T09:32:24.487 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-09T09:32:24.509 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:24 +0000 (0:00:00.049) 0:02:36.211 ********* 2021-03-09T09:32:24.638 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-09T09:32:24.640 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-09T09:32:24.711 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-09T09:32:24.728 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:24 +0000 (0:00:00.218) 0:02:36.429 ********* 2021-03-09T09:32:24.751 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:24 +0000 (0:00:00.022) 0:02:36.452 ********* 2021-03-09T09:32:25.489 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi143.front.sepia.ceph.com] 2021-03-09T09:32:25.544 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:25 +0000 (0:00:00.793) 0:02:37.245 ********* 2021-03-09T09:32:25.571 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] 2021-03-09T09:32:25.582 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:32:25.583 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-09T09:32:25.610 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:25 +0000 (0:00:00.066) 0:02:37.312 ********* 2021-03-09T09:32:25.640 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:32:25.642 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:25.669 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:25 +0000 (0:00:00.057) 0:02:37.369 ********* 2021-03-09T09:32:25.700 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:25 +0000 (0:00:00.032) 0:02:37.401 ********* 2021-03-09T09:32:25.720 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-09T09:32:25.722 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-09T09:32:25.749 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:25 +0000 (0:00:00.048) 0:02:37.450 ********* 2021-03-09T09:32:25.757 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:32:25.759 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-09T09:32:25.784 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:25 +0000 (0:00:00.034) 0:02:37.485 ********* 2021-03-09T09:32:25.811 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:32:25.813 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:25.835 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:25 +0000 (0:00:00.050) 0:02:37.536 ********* 2021-03-09T09:32:25.872 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-09T09:32:25.874 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-09T09:32:25.896 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:25 +0000 (0:00:00.061) 0:02:37.597 ********* 2021-03-09T09:32:25.898 INFO:teuthology.task.ansible.out: 2021-03-09T09:32:26.675 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-09T09:32:26.677 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"cache_update_time": 1615282174, "cache_updated": false, "changed": false} 2021-03-09T09:32:26.699 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:26 +0000 (0:00:00.803) 0:02:38.400 ********* 2021-03-09T09:32:26.727 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:32:26.729 INFO:teuthology.task.ansible.out:skipping: [smithi143.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:26.752 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:26 +0000 (0:00:00.052) 0:02:38.453 ********* 2021-03-09T09:32:26.790 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:32:26.792 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-09T09:32:26.815 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:26 +0000 (0:00:00.062) 0:02:38.516 ********* 2021-03-09T09:32:26.833 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-09T09:32:26.834 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"cache_update_time": 1615282179, "cache_updated": false, "changed": false} 2021-03-09T09:32:26.859 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:26 +0000 (0:00:00.044) 0:02:38.561 ********* 2021-03-09T09:32:26.889 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:32:26.890 INFO:teuthology.task.ansible.out:skipping: [smithi173.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T09:32:26.918 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:26 +0000 (0:00:00.058) 0:02:38.619 ********* 2021-03-09T09:32:26.956 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T09:32:26.957 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-09T09:32:26.980 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:26 +0000 (0:00:00.061) 0:02:38.681 ********* 2021-03-09T09:32:26.987 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-09T09:32:26.988 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:32:27.103 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:32:27.132 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:27 +0000 (0:00:00.151) 0:02:38.833 ********* 2021-03-09T09:32:27.140 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:32:27.255 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T09:32:27.281 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:27 +0000 (0:00:00.149) 0:02:38.982 ********* 2021-03-09T09:32:27.494 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-09T09:32:27.496 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282347.1902008-5336-6729334110976/source", "state": "file", "uid": 0} 2021-03-09T09:32:27.641 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282347.3402457-5341-216644803929794/source", "state": "file", "uid": 0} 2021-03-09T09:32:27.807 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282347.5187569-5336-86816624056249/source", "state": "file", "uid": 0} 2021-03-09T09:32:27.832 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:27 +0000 (0:00:00.551) 0:02:39.533 ********* 2021-03-09T09:32:27.952 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282347.6609797-5341-172790480459011/source", "state": "file", "uid": 0} 2021-03-09T09:32:27.979 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:27 +0000 (0:00:00.146) 0:02:39.680 ********* 2021-03-09T09:32:31.617 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T09:32:31.619 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-QlveuA-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-QlveuA-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-09T09:32:31.620 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-09T09:32:31.640 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:31 +0000 (0:00:03.661) 0:02:43.342 ********* 2021-03-09T09:32:31.771 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-09T09:32:31.772 INFO:teuthology.task.ansible.out:ok: [smithi173.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T09:32:31.797 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:31 +0000 (0:00:00.156) 0:02:43.498 ********* 2021-03-09T09:32:31.897 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T09:32:31.899 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-FjvFQj-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-FjvFQj-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-09T09:32:31.900 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-09T09:32:31.924 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:31 +0000 (0:00:00.126) 0:02:43.625 ********* 2021-03-09T09:32:32.057 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-09T09:32:32.059 INFO:teuthology.task.ansible.out:ok: [smithi143.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T09:32:32.082 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:32 +0000 (0:00:00.158) 0:02:43.784 ********* 2021-03-09T09:32:32.152 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-09T09:32:32.154 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282351.869575-5452-173116885540610/source", "state": "file", "uid": 0} 2021-03-09T09:32:32.180 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:32 +0000 (0:00:00.097) 0:02:43.881 ********* 2021-03-09T09:32:32.426 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615282352.1368563-5475-139450984721703/source", "state": "file", "uid": 0} 2021-03-09T09:32:32.449 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:32 +0000 (0:00:00.269) 0:02:44.151 ********* 2021-03-09T09:32:32.458 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-09T09:32:32.460 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.120820", "end": "2021-03-09 09:32:32.419259", "rc": 0, "start": "2021-03-09 09:32:32.298439", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:32.484 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:32 +0000 (0:00:00.034) 0:02:44.185 ********* 2021-03-09T09:32:32.708 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.118601", "end": "2021-03-09 09:32:32.791986", "rc": 0, "start": "2021-03-09 09:32:32.673385", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T09:32:32.733 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 09:32:32 +0000 (0:00:00.249) 0:02:44.435 ********* 2021-03-09T09:32:35.807 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-09T09:32:35.808 INFO:teuthology.task.ansible.out:changed: [smithi173.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "network-online.target containerd.service firewalld.service system.slice docker.socket sysinit.target basic.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-09T09:32:35.810 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-03-09T09:32:35.811 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice docker.socket", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:29:00 UTC", "StateChangeTimestampMonotonic": "41270039", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-09T09:32:35.813 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T09:32:35.969 INFO:teuthology.task.ansible.out:changed: [smithi143.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice firewalld.service network-online.target sysinit.target docker.socket systemd-journald.socket containerd.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-09T09:32:35.971 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-03-09T09:32:35.972 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice docker.socket sysinit.target", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 09:29:15 UTC", "StateChangeTimestampMonotonic": "45132783", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-09T09:32:35.974 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T09:32:36.038 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-09T09:32:36.039 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-03-09T09:32:36.041 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi143.front.sepia.ceph.com : ok=127 changed=35 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi173.front.sepia.ceph.com : ok=127 changed=35 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Tuesday 09 March 2021 09:32:36 +0000 (0:00:03.307) 0:02:47.742 ********* =============================================================================== 2021-03-09T09:32:36.041 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 20.43s users : Update authorized_keys using the keys repo ---------------------- 7.84s container-host : Install registries-conf-ctl ---------------------------- 3.66s container-host : Restart docker service --------------------------------- 3.31s testnode : Update apt cache. -------------------------------------------- 2.88s testnode : Install apt keys --------------------------------------------- 2.07s Gathering Facts --------------------------------------------------------- 1.79s testnode : Zap all non-root disks --------------------------------------- 1.31s testnode : Make sure apt dependencies are installed --------------------- 0.98s testnode : Install python-apt ------------------------------------------- 0.98s testnode : Install ntp package on apt based systems. -------------------- 0.90s testnode : Ensure packages are not present. ----------------------------- 0.90s testnode : Install python-pip on apt based systems. --------------------- 0.87s Gathering Facts --------------------------------------------------------- 0.86s common : Make sure smartmontools is installed --------------------------- 0.85s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.84s testnode : Install packages via pip ------------------------------------- 0.82s users : Clone t 2021-03-09T09:32:36.043 INFO:teuthology.task.ansible.out:he keys repo --------------------------------------------- 0.81s container-host : Install container packages ----------------------------- 0.80s Gathering Facts --------------------------------------------------------- 0.79s 2021-03-09T09:32:36.399 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi143.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2021-03-09T09:32:36.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T09:32:36.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T09:32:36.477 DEBUG:teuthology.orchestra.run.smithi143:> true 2021-03-09T09:32:36.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T09:32:36.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T09:32:36.754 DEBUG:teuthology.orchestra.run.smithi173:> true 2021-03-09T09:32:36.943 INFO:teuthology.run_tasks:Running task clock... 2021-03-09T09:32:36.956 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-09T09:32:36.957 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T09:32:36.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T09:32:37.032 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-09T09:32:37.033 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: Command line: ntpd -gq 2021-03-09T09:32:37.033 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: proto: precision = 0.124 usec (-23) 2021-03-09T09:32:37.034 INFO:teuthology.orchestra.run.smithi143.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T09:32:37.035 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T09:32:37.036 INFO:teuthology.orchestra.run.smithi143.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-09T09:32:37.037 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: restrict ::: KOD does nothing without LIMITED. 2021-03-09T09:32:37.037 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: Listen and drop on 0 v6wildcard [::]:123 2021-03-09T09:32:37.037 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-09T09:32:37.038 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: Listen normally on 2 lo 127.0.0.1:123 2021-03-09T09:32:37.038 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: Listen normally on 3 enp3s0f1 172.21.15.143:123 2021-03-09T09:32:37.038 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: Listen normally on 4 lo [::1]:123 2021-03-09T09:32:37.038 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7783%5]:123 2021-03-09T09:32:37.039 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:37 ntpd[9240]: Listening on routing socket on fd #22 for interface updates 2021-03-09T09:32:38.034 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:38 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:38.034 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:38 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:38.034 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:38 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:38.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:38 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:38.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:38 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:38.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:38 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:38.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:38 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:38.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:38 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:40.034 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:40.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:40.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:40.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:40.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:40.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:40.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:40.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:40.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:40.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:40.037 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:40.037 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:40 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:42.034 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:42.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:42.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:42.035 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:42.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:42.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:42.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:42.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:42.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:42.037 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:42.037 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:42.037 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:42.037 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:42.038 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:42.038 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:42.038 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:42 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:44.033 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:44 ntpd[9240]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:44.034 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:44 ntpd[9240]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:44.034 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:44 ntpd[9240]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:44.034 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:44 ntpd[9240]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:44.035 INFO:teuthology.orchestra.run.smithi143.stdout: 9 Mar 09:32:44 ntpd[9240]: ntpd: time slew +0.000314 s 2021-03-09T09:32:44.035 INFO:teuthology.orchestra.run.smithi143.stdout:ntpd: time slew +0.000314s 2021-03-09T09:32:44.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:44 ntpd[9240]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-09T09:32:44.036 INFO:teuthology.orchestra.run.smithi143.stderr: 9 Mar 09:32:44 ntpd[9240]: can't open /var/log/ntpstats/loopstats.20210309: Permission denied 2021-03-09T09:32:44.174 INFO:teuthology.orchestra.run.smithi143.stdout: remote refid st t when poll reach delay offset jitter 2021-03-09T09:32:44.174 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================== 2021-03-09T09:32:44.174 INFO:teuthology.orchestra.run.smithi143.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:32:44.175 INFO:teuthology.orchestra.run.smithi143.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:32:44.175 INFO:teuthology.orchestra.run.smithi143.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:32:44.175 INFO:teuthology.orchestra.run.smithi143.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:32:44.176 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T09:32:44.176 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T09:32:44.239 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-09T09:32:44.239 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: Command line: ntpd -gq 2021-03-09T09:32:44.240 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: proto: precision = 0.123 usec (-23) 2021-03-09T09:32:44.240 INFO:teuthology.orchestra.run.smithi173.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T09:32:44.241 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-09T09:32:44.241 INFO:teuthology.orchestra.run.smithi173.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-09T09:32:44.243 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: restrict ::: KOD does nothing without LIMITED. 2021-03-09T09:32:44.243 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: Listen and drop on 0 v6wildcard [::]:123 2021-03-09T09:32:44.244 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-09T09:32:44.244 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: Listen normally on 2 lo 127.0.0.1:123 2021-03-09T09:32:44.245 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: Listen normally on 3 enp3s0f1 172.21.15.173:123 2021-03-09T09:32:44.245 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: Listen normally on 4 lo [::1]:123 2021-03-09T09:32:44.245 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7259%5]:123 2021-03-09T09:32:44.246 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:44 ntpd[9254]: Listening on routing socket on fd #22 for interface updates 2021-03-09T09:32:45.240 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:45 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:45.240 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:45 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:45.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:45 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:45.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:45 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:45.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:45 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:45.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:45 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:45.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:45 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:45.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:45 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:47.240 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:47.240 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:47.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:47.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:47.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:47.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:47.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:47.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:47.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:47.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:47.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:47.243 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:47 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:49.240 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:49.240 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:49.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:49.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:49.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:49.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:49.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:49.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:49.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:49.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:49.243 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:49.243 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:49.243 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:49.243 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:49.243 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:49.244 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:49 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:51.239 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:51 ntpd[9254]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-09T09:32:51.240 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:51 ntpd[9254]: can't open /var/log/ntpstats/rawstats.20210309: Permission denied 2021-03-09T09:32:51.240 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:51 ntpd[9254]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-09T09:32:51.240 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:51 ntpd[9254]: can't open /var/log/ntpstats/peerstats.20210309: Permission denied 2021-03-09T09:32:51.241 INFO:teuthology.orchestra.run.smithi173.stdout: 9 Mar 09:32:51 ntpd[9254]: ntpd: time slew +0.000584 s 2021-03-09T09:32:51.241 INFO:teuthology.orchestra.run.smithi173.stdout:ntpd: time slew +0.000584s 2021-03-09T09:32:51.241 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:51 ntpd[9254]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-09T09:32:51.242 INFO:teuthology.orchestra.run.smithi173.stderr: 9 Mar 09:32:51 ntpd[9254]: can't open /var/log/ntpstats/loopstats.20210309: Permission denied 2021-03-09T09:32:51.292 INFO:teuthology.orchestra.run.smithi173.stdout: remote refid st t when poll reach delay offset jitter 2021-03-09T09:32:51.293 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================== 2021-03-09T09:32:51.293 INFO:teuthology.orchestra.run.smithi173.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:32:51.294 INFO:teuthology.orchestra.run.smithi173.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:32:51.294 INFO:teuthology.orchestra.run.smithi173.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:32:51.294 INFO:teuthology.orchestra.run.smithi173.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-09T09:32:51.294 INFO:teuthology.run_tasks:Running task install... 2021-03-09T09:32:51.303 DEBUG:teuthology.task.install:project ceph 2021-03-09T09:32:51.304 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}} 2021-03-09T09:32:51.304 DEBUG:teuthology.task.install:config {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'} 2021-03-09T09:32:51.305 INFO:teuthology.task.install:Using flavor: basic 2021-03-09T09:32:51.313 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-09T09:32:51.314 INFO:teuthology.task.install:extra packages: [] 2021-03-09T09:32:51.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo lsb_release -is 2021-03-09T09:32:51.367 INFO:teuthology.orchestra.run.smithi143.stdout:Ubuntu 2021-03-09T09:32:51.373 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:32:51.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo lsb_release -is 2021-03-09T09:32:51.374 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-key list | grep Ceph 2021-03-09T09:32:51.427 INFO:teuthology.orchestra.run.smithi173.stdout:Ubuntu 2021-03-09T09:32:51.432 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:32:51.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-key list | grep Ceph 2021-03-09T09:32:51.461 INFO:teuthology.orchestra.run.smithi143.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-09T09:32:51.520 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-09T09:32:51.520 INFO:teuthology.orchestra.run.smithi143.stdout:uid [ unknown] Ceph.com (release key) 2021-03-09T09:32:51.521 DEBUG:teuthology.orchestra.run.smithi143:> sudo lsb_release -is 2021-03-09T09:32:51.525 INFO:teuthology.orchestra.run.smithi173.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-09T09:32:51.575 INFO:teuthology.orchestra.run.smithi143.stdout:Ubuntu 2021-03-09T09:32:51.579 INFO:teuthology.orchestra.run.smithi173.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-09T09:32:51.579 INFO:teuthology.orchestra.run.smithi173.stdout:uid [ unknown] Ceph.com (release key) 2021-03-09T09:32:51.580 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:32:51.581 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-09T09:32:51.581 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:32:51.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo lsb_release -is 2021-03-09T09:32:51.636 INFO:teuthology.orchestra.run.smithi173.stdout:Ubuntu 2021-03-09T09:32:51.641 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:32:51.641 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-09T09:32:51.642 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:32:51.741 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default/ 2021-03-09T09:32:51.741 INFO:teuthology.task.install.deb:Package version is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:32:51.783 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default/ 2021-03-09T09:32:51.784 INFO:teuthology.task.install.deb:Package version is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:32:51.858 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:32:51.859 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-09T09:32:51.874 DEBUG:teuthology.orchestra.run.smithi143:> sudo apt-get update 2021-03-09T09:32:51.899 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:32:51.900 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-09T09:32:51.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo apt-get update 2021-03-09T09:32:52.096 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-09T09:32:52.130 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic InRelease 2021-03-09T09:32:52.153 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-09T09:32:52.159 INFO:teuthology.orchestra.run.smithi173.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic InRelease 2021-03-09T09:32:52.177 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-09T09:32:52.178 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T09:32:52.179 INFO:teuthology.orchestra.run.smithi143.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-09T09:32:52.182 INFO:teuthology.orchestra.run.smithi173.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-09T09:32:52.202 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,099 B] 2021-03-09T09:32:52.208 INFO:teuthology.orchestra.run.smithi173.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-09T09:32:52.210 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-09T09:32:52.231 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-03-09T09:32:52.235 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.6 kB] 2021-03-09T09:32:52.251 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-09T09:32:52.263 INFO:teuthology.orchestra.run.smithi173.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,099 B] 2021-03-09T09:32:52.289 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-09T09:32:52.330 INFO:teuthology.orchestra.run.smithi143.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-09T09:32:52.369 INFO:teuthology.orchestra.run.smithi173.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-09T09:32:52.703 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 27.4 kB in 1s (46.6 kB/s) 2021-03-09T09:32:52.787 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 27.4 kB in 1s (38.2 kB/s) 2021-03-09T09:32:53.954 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2021-03-09T09:32:54.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-1742-g89fb622b-1bionic cephadm=17.0.0-1742-g89fb622b-1bionic ceph-mds=17.0.0-1742-g89fb622b-1bionic ceph-mgr=17.0.0-1742-g89fb622b-1bionic ceph-common=17.0.0-1742-g89fb622b-1bionic ceph-fuse=17.0.0-1742-g89fb622b-1bionic ceph-test=17.0.0-1742-g89fb622b-1bionic radosgw=17.0.0-1742-g89fb622b-1bionic python3-rados=17.0.0-1742-g89fb622b-1bionic python3-rgw=17.0.0-1742-g89fb622b-1bionic python3-cephfs=17.0.0-1742-g89fb622b-1bionic python3-rbd=17.0.0-1742-g89fb622b-1bionic libcephfs2=17.0.0-1742-g89fb622b-1bionic libcephfs-dev=17.0.0-1742-g89fb622b-1bionic librados2=17.0.0-1742-g89fb622b-1bionic librbd1=17.0.0-1742-g89fb622b-1bionic rbd-fuse=17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:32:54.037 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2021-03-09T09:32:54.081 INFO:teuthology.orchestra.run.smithi173.stdout:Reading package lists... 2021-03-09T09:32:54.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-1742-g89fb622b-1bionic cephadm=17.0.0-1742-g89fb622b-1bionic ceph-mds=17.0.0-1742-g89fb622b-1bionic ceph-mgr=17.0.0-1742-g89fb622b-1bionic ceph-common=17.0.0-1742-g89fb622b-1bionic ceph-fuse=17.0.0-1742-g89fb622b-1bionic ceph-test=17.0.0-1742-g89fb622b-1bionic radosgw=17.0.0-1742-g89fb622b-1bionic python3-rados=17.0.0-1742-g89fb622b-1bionic python3-rgw=17.0.0-1742-g89fb622b-1bionic python3-cephfs=17.0.0-1742-g89fb622b-1bionic python3-rbd=17.0.0-1742-g89fb622b-1bionic libcephfs2=17.0.0-1742-g89fb622b-1bionic libcephfs-dev=17.0.0-1742-g89fb622b-1bionic librados2=17.0.0-1742-g89fb622b-1bionic librbd1=17.0.0-1742-g89fb622b-1bionic rbd-fuse=17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:32:54.162 INFO:teuthology.orchestra.run.smithi143.stdout:Reading package lists... 2021-03-09T09:32:54.312 INFO:teuthology.orchestra.run.smithi173.stdout:Building dependency tree... 2021-03-09T09:32:54.313 INFO:teuthology.orchestra.run.smithi173.stdout:Reading state information... 2021-03-09T09:32:54.434 INFO:teuthology.orchestra.run.smithi143.stdout:Building dependency tree... 2021-03-09T09:32:54.435 INFO:teuthology.orchestra.run.smithi143.stdout:Reading state information... 2021-03-09T09:32:54.481 INFO:teuthology.orchestra.run.smithi173.stdout:The following additional packages will be installed: 2021-03-09T09:32:54.481 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-09T09:32:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-09T09:32:54.482 INFO:teuthology.orchestra.run.smithi173.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-09T09:32:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-09T09:32:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-09T09:32:54.483 INFO:teuthology.orchestra.run.smithi173.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T09:32:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-09T09:32:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-09T09:32:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-09T09:32:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-09T09:32:54.484 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-09T09:32:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-09T09:32:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-09T09:32:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-09T09:32:54.485 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-09T09:32:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-09T09:32:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-09T09:32:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-09T09:32:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-09T09:32:54.486 INFO:teuthology.orchestra.run.smithi173.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-09T09:32:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-09T09:32:54.487 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-09T09:32:54.488 INFO:teuthology.orchestra.run.smithi173.stdout:Suggested packages: 2021-03-09T09:32:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-09T09:32:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-09T09:32:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-09T09:32:54.488 INFO:teuthology.orchestra.run.smithi173.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-09T09:32:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-09T09:32:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-09T09:32:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-09T09:32:54.489 INFO:teuthology.orchestra.run.smithi173.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-09T09:32:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-09T09:32:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-09T09:32:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-09T09:32:54.490 INFO:teuthology.orchestra.run.smithi173.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-09T09:32:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-09T09:32:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-09T09:32:54.491 INFO:teuthology.orchestra.run.smithi173.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-09T09:32:54.585 INFO:teuthology.orchestra.run.smithi173.stdout:The following NEW packages will be installed: 2021-03-09T09:32:54.585 INFO:teuthology.orchestra.run.smithi173.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-09T09:32:54.586 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-09T09:32:54.586 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-09T09:32:54.587 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-03-09T09:32:54.587 INFO:teuthology.orchestra.run.smithi173.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-09T09:32:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-09T09:32:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-09T09:32:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T09:32:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-09T09:32:54.588 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-09T09:32:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-09T09:32:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-09T09:32:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-09T09:32:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-09T09:32:54.589 INFO:teuthology.orchestra.run.smithi173.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-09T09:32:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-09T09:32:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-09T09:32:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-09T09:32:54.590 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-09T09:32:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-09T09:32:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-09T09:32:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-09T09:32:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-09T09:32:54.591 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-09T09:32:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-09T09:32:54.592 INFO:teuthology.orchestra.run.smithi173.stdout:The following packages will be upgraded: 2021-03-09T09:32:54.592 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 librbd1 2021-03-09T09:32:54.608 INFO:teuthology.orchestra.run.smithi143.stdout:The following additional packages will be installed: 2021-03-09T09:32:54.608 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-03-09T09:32:54.609 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-03-09T09:32:54.609 INFO:teuthology.orchestra.run.smithi143.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-09T09:32:54.610 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-09T09:32:54.610 INFO:teuthology.orchestra.run.smithi143.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-09T09:32:54.610 INFO:teuthology.orchestra.run.smithi143.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T09:32:54.610 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-09T09:32:54.610 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-09T09:32:54.611 INFO:teuthology.orchestra.run.smithi143.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-09T09:32:54.611 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-09T09:32:54.611 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-09T09:32:54.611 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-09T09:32:54.611 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-09T09:32:54.612 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-09T09:32:54.612 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-09T09:32:54.612 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-09T09:32:54.612 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-09T09:32:54.613 INFO:teuthology.orchestra.run.smithi143.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-09T09:32:54.613 INFO:teuthology.orchestra.run.smithi143.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-09T09:32:54.613 INFO:teuthology.orchestra.run.smithi143.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-09T09:32:54.613 INFO:teuthology.orchestra.run.smithi143.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-09T09:32:54.613 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-09T09:32:54.614 INFO:teuthology.orchestra.run.smithi143.stdout:Suggested packages: 2021-03-09T09:32:54.614 INFO:teuthology.orchestra.run.smithi143.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-09T09:32:54.614 INFO:teuthology.orchestra.run.smithi143.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-09T09:32:54.615 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-09T09:32:54.615 INFO:teuthology.orchestra.run.smithi143.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-09T09:32:54.615 INFO:teuthology.orchestra.run.smithi143.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-09T09:32:54.615 INFO:teuthology.orchestra.run.smithi143.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-09T09:32:54.616 INFO:teuthology.orchestra.run.smithi143.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-09T09:32:54.616 INFO:teuthology.orchestra.run.smithi143.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-09T09:32:54.616 INFO:teuthology.orchestra.run.smithi143.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-09T09:32:54.616 INFO:teuthology.orchestra.run.smithi143.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-09T09:32:54.616 INFO:teuthology.orchestra.run.smithi143.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-09T09:32:54.617 INFO:teuthology.orchestra.run.smithi143.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-09T09:32:54.617 INFO:teuthology.orchestra.run.smithi143.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-09T09:32:54.617 INFO:teuthology.orchestra.run.smithi143.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-09T09:32:54.617 INFO:teuthology.orchestra.run.smithi143.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-09T09:32:54.716 INFO:teuthology.orchestra.run.smithi143.stdout:The following NEW packages will be installed: 2021-03-09T09:32:54.716 INFO:teuthology.orchestra.run.smithi143.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-09T09:32:54.717 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-03-09T09:32:54.718 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-03-09T09:32:54.718 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-03-09T09:32:54.718 INFO:teuthology.orchestra.run.smithi143.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-09T09:32:54.719 INFO:teuthology.orchestra.run.smithi143.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-09T09:32:54.719 INFO:teuthology.orchestra.run.smithi143.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-09T09:32:54.719 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-09T09:32:54.719 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-09T09:32:54.719 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-09T09:32:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-09T09:32:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-09T09:32:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-09T09:32:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-09T09:32:54.720 INFO:teuthology.orchestra.run.smithi143.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-09T09:32:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-09T09:32:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-09T09:32:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-09T09:32:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-09T09:32:54.721 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-09T09:32:54.722 INFO:teuthology.orchestra.run.smithi143.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-09T09:32:54.722 INFO:teuthology.orchestra.run.smithi143.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-09T09:32:54.722 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-09T09:32:54.722 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-09T09:32:54.723 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-09T09:32:54.724 INFO:teuthology.orchestra.run.smithi173.stdout:2 upgraded, 111 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T09:32:54.724 INFO:teuthology.orchestra.run.smithi173.stdout:Need to get 166 MB of archives. 2021-03-09T09:32:54.724 INFO:teuthology.orchestra.run.smithi173.stdout:After this operation, 778 MB of additional disk space will be used. 2021-03-09T09:32:54.724 INFO:teuthology.orchestra.run.smithi173.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-1742-g89fb622b-1bionic [3,728 B] 2021-03-09T09:32:54.725 INFO:teuthology.orchestra.run.smithi143.stdout:The following packages will be upgraded: 2021-03-09T09:32:54.725 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 librbd1 2021-03-09T09:32:54.769 INFO:teuthology.orchestra.run.smithi173.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-1742-g89fb622b-1bionic [3,126 kB] 2021-03-09T09:32:54.772 INFO:teuthology.orchestra.run.smithi173.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-09T09:32:54.851 INFO:teuthology.orchestra.run.smithi173.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-09T09:32:54.853 INFO:teuthology.orchestra.run.smithi143.stdout:2 upgraded, 111 newly installed, 0 to remove and 15 not upgraded. 2021-03-09T09:32:54.853 INFO:teuthology.orchestra.run.smithi143.stdout:Need to get 166 MB of archives. 2021-03-09T09:32:54.853 INFO:teuthology.orchestra.run.smithi143.stdout:After this operation, 778 MB of additional disk space will be used. 2021-03-09T09:32:54.854 INFO:teuthology.orchestra.run.smithi143.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 17.0.0-1742-g89fb622b-1bionic [3,728 B] 2021-03-09T09:32:54.899 INFO:teuthology.orchestra.run.smithi143.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 17.0.0-1742-g89fb622b-1bionic [3,126 kB] 2021-03-09T09:32:54.910 INFO:teuthology.orchestra.run.smithi143.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-09T09:32:54.964 INFO:teuthology.orchestra.run.smithi173.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-1742-g89fb622b-1bionic [3,169 kB] 2021-03-09T09:32:54.991 INFO:teuthology.orchestra.run.smithi143.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-09T09:32:55.074 INFO:teuthology.orchestra.run.smithi173.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-1742-g89fb622b-1bionic [665 kB] 2021-03-09T09:32:55.095 INFO:teuthology.orchestra.run.smithi173.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-1742-g89fb622b-1bionic [340 kB] 2021-03-09T09:32:55.096 INFO:teuthology.orchestra.run.smithi173.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-09T09:32:55.106 INFO:teuthology.orchestra.run.smithi173.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-09T09:32:55.106 INFO:teuthology.orchestra.run.smithi173.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-1742-g89fb622b-1bionic [21.7 kB] 2021-03-09T09:32:55.107 INFO:teuthology.orchestra.run.smithi173.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-1742-g89fb622b-1bionic [163 kB] 2021-03-09T09:32:55.112 INFO:teuthology.orchestra.run.smithi173.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-09T09:32:55.115 INFO:teuthology.orchestra.run.smithi173.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-1742-g89fb622b-1bionic [29.4 kB] 2021-03-09T09:32:55.117 INFO:teuthology.orchestra.run.smithi173.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-1742-g89fb622b-1bionic [336 kB] 2021-03-09T09:32:55.128 INFO:teuthology.orchestra.run.smithi173.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-1742-g89fb622b-1bionic [3,311 kB] 2021-03-09T09:32:55.210 INFO:teuthology.orchestra.run.smithi173.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-09T09:32:55.213 INFO:teuthology.orchestra.run.smithi173.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-09T09:32:55.239 INFO:teuthology.orchestra.run.smithi173.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-1742-g89fb622b-1bionic [99.3 kB] 2021-03-09T09:32:55.241 INFO:teuthology.orchestra.run.smithi143.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-09T09:32:55.242 INFO:teuthology.orchestra.run.smithi143.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-09T09:32:55.245 INFO:teuthology.orchestra.run.smithi173.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-1742-g89fb622b-1bionic [414 kB] 2021-03-09T09:32:55.247 INFO:teuthology.orchestra.run.smithi143.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-09T09:32:55.259 INFO:teuthology.orchestra.run.smithi173.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-1742-g89fb622b-1bionic [21.2 MB] 2021-03-09T09:32:55.261 INFO:teuthology.orchestra.run.smithi173.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-09T09:32:55.268 INFO:teuthology.orchestra.run.smithi173.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-09T09:32:55.276 INFO:teuthology.orchestra.run.smithi173.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-09T09:32:55.287 INFO:teuthology.orchestra.run.smithi173.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-09T09:32:55.294 INFO:teuthology.orchestra.run.smithi173.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-09T09:32:55.312 INFO:teuthology.orchestra.run.smithi173.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-09T09:32:55.315 INFO:teuthology.orchestra.run.smithi173.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-09T09:32:55.332 INFO:teuthology.orchestra.run.smithi173.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-09T09:32:55.360 INFO:teuthology.orchestra.run.smithi143.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-09T09:32:55.362 INFO:teuthology.orchestra.run.smithi173.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-09T09:32:55.363 INFO:teuthology.orchestra.run.smithi143.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-09T09:32:55.400 INFO:teuthology.orchestra.run.smithi143.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-09T09:32:55.410 INFO:teuthology.orchestra.run.smithi173.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-09T09:32:55.412 INFO:teuthology.orchestra.run.smithi173.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-09T09:32:55.412 INFO:teuthology.orchestra.run.smithi173.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-09T09:32:55.414 INFO:teuthology.orchestra.run.smithi173.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-09T09:32:55.415 INFO:teuthology.orchestra.run.smithi173.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-09T09:32:55.416 INFO:teuthology.orchestra.run.smithi173.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-09T09:32:55.418 INFO:teuthology.orchestra.run.smithi173.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-09T09:32:55.419 INFO:teuthology.orchestra.run.smithi143.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-09T09:32:55.421 INFO:teuthology.orchestra.run.smithi173.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-09T09:32:55.428 INFO:teuthology.orchestra.run.smithi143.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-09T09:32:55.445 INFO:teuthology.orchestra.run.smithi143.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-09T09:32:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-09T09:32:55.472 INFO:teuthology.orchestra.run.smithi143.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-09T09:32:55.476 INFO:teuthology.orchestra.run.smithi143.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-09T09:32:55.483 INFO:teuthology.orchestra.run.smithi143.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-09T09:32:55.489 INFO:teuthology.orchestra.run.smithi173.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-09T09:32:55.490 INFO:teuthology.orchestra.run.smithi173.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-09T09:32:55.505 INFO:teuthology.orchestra.run.smithi143.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-09T09:32:55.506 INFO:teuthology.orchestra.run.smithi143.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-09T09:32:55.563 INFO:teuthology.orchestra.run.smithi143.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-09T09:32:55.564 INFO:teuthology.orchestra.run.smithi143.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-09T09:32:55.567 INFO:teuthology.orchestra.run.smithi143.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-09T09:32:55.567 INFO:teuthology.orchestra.run.smithi173.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-09T09:32:55.568 INFO:teuthology.orchestra.run.smithi173.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-09T09:32:55.569 INFO:teuthology.orchestra.run.smithi143.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-09T09:32:55.569 INFO:teuthology.orchestra.run.smithi143.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-09T09:32:55.569 INFO:teuthology.orchestra.run.smithi173.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-09T09:32:55.570 INFO:teuthology.orchestra.run.smithi173.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-09T09:32:55.571 INFO:teuthology.orchestra.run.smithi143.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-09T09:32:55.573 INFO:teuthology.orchestra.run.smithi143.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-09T09:32:55.573 INFO:teuthology.orchestra.run.smithi173.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-09T09:32:55.581 INFO:teuthology.orchestra.run.smithi173.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-09T09:32:55.585 INFO:teuthology.orchestra.run.smithi173.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-09T09:32:55.586 INFO:teuthology.orchestra.run.smithi173.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-09T09:32:55.643 INFO:teuthology.orchestra.run.smithi143.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-09T09:32:55.644 INFO:teuthology.orchestra.run.smithi143.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-09T09:32:55.646 INFO:teuthology.orchestra.run.smithi173.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-09T09:32:55.664 INFO:teuthology.orchestra.run.smithi143.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-09T09:32:55.723 INFO:teuthology.orchestra.run.smithi143.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-09T09:32:55.734 INFO:teuthology.orchestra.run.smithi143.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-09T09:32:55.735 INFO:teuthology.orchestra.run.smithi143.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-09T09:32:55.736 INFO:teuthology.orchestra.run.smithi173.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-09T09:32:55.737 INFO:teuthology.orchestra.run.smithi173.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-09T09:32:55.738 INFO:teuthology.orchestra.run.smithi143.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-09T09:32:55.747 INFO:teuthology.orchestra.run.smithi143.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-09T09:32:55.751 INFO:teuthology.orchestra.run.smithi143.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-09T09:32:55.751 INFO:teuthology.orchestra.run.smithi143.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-09T09:32:55.803 INFO:teuthology.orchestra.run.smithi143.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-09T09:32:55.890 INFO:teuthology.orchestra.run.smithi143.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-09T09:32:55.891 INFO:teuthology.orchestra.run.smithi143.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-09T09:32:56.014 INFO:teuthology.orchestra.run.smithi173.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-1742-g89fb622b-1bionic [5,646 kB] 2021-03-09T09:32:56.014 INFO:teuthology.orchestra.run.smithi173.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-09T09:32:56.049 INFO:teuthology.orchestra.run.smithi173.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-09T09:32:56.051 INFO:teuthology.orchestra.run.smithi173.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-09T09:32:56.093 INFO:teuthology.orchestra.run.smithi173.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-09T09:32:56.093 INFO:teuthology.orchestra.run.smithi173.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-09T09:32:56.093 INFO:teuthology.orchestra.run.smithi173.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-09T09:32:56.094 INFO:teuthology.orchestra.run.smithi173.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-09T09:32:56.095 INFO:teuthology.orchestra.run.smithi173.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-09T09:32:56.096 INFO:teuthology.orchestra.run.smithi173.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-09T09:32:56.097 INFO:teuthology.orchestra.run.smithi173.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-09T09:32:56.172 INFO:teuthology.orchestra.run.smithi173.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-09T09:32:56.174 INFO:teuthology.orchestra.run.smithi143.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-09T09:32:56.174 INFO:teuthology.orchestra.run.smithi173.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-09T09:32:56.181 INFO:teuthology.orchestra.run.smithi173.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-09T09:32:56.181 INFO:teuthology.orchestra.run.smithi173.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-09T09:32:56.203 INFO:teuthology.orchestra.run.smithi143.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-09T09:32:56.205 INFO:teuthology.orchestra.run.smithi143.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-09T09:32:56.216 INFO:teuthology.orchestra.run.smithi173.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-1742-g89fb622b-1bionic [184 kB] 2021-03-09T09:32:56.221 INFO:teuthology.orchestra.run.smithi173.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-1742-g89fb622b-1bionic [1,388 kB] 2021-03-09T09:32:56.251 INFO:teuthology.orchestra.run.smithi173.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-09T09:32:56.251 INFO:teuthology.orchestra.run.smithi173.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-09T09:32:56.252 INFO:teuthology.orchestra.run.smithi173.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-09T09:32:56.254 INFO:teuthology.orchestra.run.smithi143.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-09T09:32:56.261 INFO:teuthology.orchestra.run.smithi143.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-09T09:32:56.261 INFO:teuthology.orchestra.run.smithi143.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-09T09:32:56.261 INFO:teuthology.orchestra.run.smithi143.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-09T09:32:56.261 INFO:teuthology.orchestra.run.smithi143.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-09T09:32:56.262 INFO:teuthology.orchestra.run.smithi143.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-09T09:32:56.262 INFO:teuthology.orchestra.run.smithi143.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-09T09:32:56.263 INFO:teuthology.orchestra.run.smithi173.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-09T09:32:56.263 INFO:teuthology.orchestra.run.smithi173.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-09T09:32:56.263 INFO:teuthology.orchestra.run.smithi173.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-09T09:32:56.274 INFO:teuthology.orchestra.run.smithi173.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-1742-g89fb622b-1bionic [6,670 kB] 2021-03-09T09:32:56.329 INFO:teuthology.orchestra.run.smithi173.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-09T09:32:56.334 INFO:teuthology.orchestra.run.smithi143.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-09T09:32:56.335 INFO:teuthology.orchestra.run.smithi173.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-09T09:32:56.336 INFO:teuthology.orchestra.run.smithi143.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-09T09:32:56.341 INFO:teuthology.orchestra.run.smithi173.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-09T09:32:56.344 INFO:teuthology.orchestra.run.smithi173.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-09T09:32:56.367 INFO:teuthology.orchestra.run.smithi143.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 17.0.0-1742-g89fb622b-1bionic [3,169 kB] 2021-03-09T09:32:56.407 INFO:teuthology.orchestra.run.smithi173.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-09T09:32:56.408 INFO:teuthology.orchestra.run.smithi173.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-09T09:32:56.409 INFO:teuthology.orchestra.run.smithi173.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-09T09:32:56.411 INFO:teuthology.orchestra.run.smithi173.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-09T09:32:56.415 INFO:teuthology.orchestra.run.smithi173.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-09T09:32:56.416 INFO:teuthology.orchestra.run.smithi173.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-09T09:32:56.420 INFO:teuthology.orchestra.run.smithi143.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-09T09:32:56.421 INFO:teuthology.orchestra.run.smithi143.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-09T09:32:56.426 INFO:teuthology.orchestra.run.smithi143.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-09T09:32:56.426 INFO:teuthology.orchestra.run.smithi143.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-09T09:32:56.427 INFO:teuthology.orchestra.run.smithi143.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-09T09:32:56.430 INFO:teuthology.orchestra.run.smithi143.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-09T09:32:56.433 INFO:teuthology.orchestra.run.smithi143.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-09T09:32:56.433 INFO:teuthology.orchestra.run.smithi143.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-09T09:32:56.486 INFO:teuthology.orchestra.run.smithi173.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-09T09:32:56.486 INFO:teuthology.orchestra.run.smithi173.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-09T09:32:56.489 INFO:teuthology.orchestra.run.smithi173.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-09T09:32:56.500 INFO:teuthology.orchestra.run.smithi143.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-09T09:32:56.506 INFO:teuthology.orchestra.run.smithi143.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-09T09:32:56.508 INFO:teuthology.orchestra.run.smithi173.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-09T09:32:56.525 INFO:teuthology.orchestra.run.smithi173.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-1742-g89fb622b-1bionic [24.9 MB] 2021-03-09T09:32:56.564 INFO:teuthology.orchestra.run.smithi173.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-09T09:32:56.565 INFO:teuthology.orchestra.run.smithi173.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-09T09:32:56.580 INFO:teuthology.orchestra.run.smithi143.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-09T09:32:56.587 INFO:teuthology.orchestra.run.smithi143.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-09T09:32:56.588 INFO:teuthology.orchestra.run.smithi143.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-09T09:32:56.589 INFO:teuthology.orchestra.run.smithi143.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-09T09:32:56.590 INFO:teuthology.orchestra.run.smithi143.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-09T09:32:56.591 INFO:teuthology.orchestra.run.smithi143.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-09T09:32:56.594 INFO:teuthology.orchestra.run.smithi143.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-09T09:32:56.594 INFO:teuthology.orchestra.run.smithi143.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-09T09:32:56.660 INFO:teuthology.orchestra.run.smithi143.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-09T09:32:56.660 INFO:teuthology.orchestra.run.smithi143.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-09T09:32:56.663 INFO:teuthology.orchestra.run.smithi173.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-09T09:32:56.663 INFO:teuthology.orchestra.run.smithi173.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-09T09:32:56.664 INFO:teuthology.orchestra.run.smithi173.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-03-09T09:32:56.668 INFO:teuthology.orchestra.run.smithi173.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-09T09:32:56.668 INFO:teuthology.orchestra.run.smithi173.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-09T09:32:56.669 INFO:teuthology.orchestra.run.smithi173.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-09T09:32:56.669 INFO:teuthology.orchestra.run.smithi173.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-09T09:32:56.670 INFO:teuthology.orchestra.run.smithi173.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-09T09:32:56.720 INFO:teuthology.orchestra.run.smithi173.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-09T09:32:56.739 INFO:teuthology.orchestra.run.smithi143.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-09T09:32:56.762 INFO:teuthology.orchestra.run.smithi143.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-09T09:32:56.763 INFO:teuthology.orchestra.run.smithi143.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-09T09:32:56.763 INFO:teuthology.orchestra.run.smithi143.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-09T09:32:56.915 INFO:teuthology.orchestra.run.smithi143.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-09T09:32:56.915 INFO:teuthology.orchestra.run.smithi143.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-09T09:32:56.916 INFO:teuthology.orchestra.run.smithi143.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-03-09T09:32:56.923 INFO:teuthology.orchestra.run.smithi143.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-09T09:32:56.923 INFO:teuthology.orchestra.run.smithi143.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-09T09:32:56.924 INFO:teuthology.orchestra.run.smithi143.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-09T09:32:56.924 INFO:teuthology.orchestra.run.smithi143.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-09T09:32:56.926 INFO:teuthology.orchestra.run.smithi143.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-09T09:32:56.926 INFO:teuthology.orchestra.run.smithi143.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-09T09:32:57.108 INFO:teuthology.orchestra.run.smithi143.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 17.0.0-1742-g89fb622b-1bionic [665 kB] 2021-03-09T09:32:57.260 INFO:teuthology.orchestra.run.smithi143.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 17.0.0-1742-g89fb622b-1bionic [340 kB] 2021-03-09T09:32:57.335 INFO:teuthology.orchestra.run.smithi143.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 17.0.0-1742-g89fb622b-1bionic [21.7 kB] 2021-03-09T09:32:57.338 INFO:teuthology.orchestra.run.smithi143.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 17.0.0-1742-g89fb622b-1bionic [163 kB] 2021-03-09T09:32:57.344 INFO:teuthology.orchestra.run.smithi143.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 17.0.0-1742-g89fb622b-1bionic [29.4 kB] 2021-03-09T09:32:57.346 INFO:teuthology.orchestra.run.smithi143.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 17.0.0-1742-g89fb622b-1bionic [336 kB] 2021-03-09T09:32:57.564 INFO:teuthology.orchestra.run.smithi173.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-1742-g89fb622b-1bionic [3,812 B] 2021-03-09T09:32:57.564 INFO:teuthology.orchestra.run.smithi173.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-1742-g89fb622b-1bionic [771 kB] 2021-03-09T09:32:57.565 INFO:teuthology.orchestra.run.smithi143.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 17.0.0-1742-g89fb622b-1bionic [3,311 kB] 2021-03-09T09:32:57.571 INFO:teuthology.orchestra.run.smithi173.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-1742-g89fb622b-1bionic [2,153 kB] 2021-03-09T09:32:57.594 INFO:teuthology.orchestra.run.smithi173.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-1742-g89fb622b-1bionic [63.1 kB] 2021-03-09T09:32:57.596 INFO:teuthology.orchestra.run.smithi173.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-1742-g89fb622b-1bionic [68.9 kB] 2021-03-09T09:32:57.598 INFO:teuthology.orchestra.run.smithi173.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-1742-g89fb622b-1bionic [2,333 kB] 2021-03-09T09:32:57.697 INFO:teuthology.orchestra.run.smithi173.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-1742-g89fb622b-1bionic [7,881 kB] 2021-03-09T09:32:58.013 INFO:teuthology.orchestra.run.smithi173.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-1742-g89fb622b-1bionic [3,928 B] 2021-03-09T09:32:58.013 INFO:teuthology.orchestra.run.smithi173.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-1742-g89fb622b-1bionic [22.5 kB] 2021-03-09T09:32:58.014 INFO:teuthology.orchestra.run.smithi173.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-1742-g89fb622b-1bionic [45.2 MB] 2021-03-09T09:32:58.028 INFO:teuthology.orchestra.run.smithi143.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 17.0.0-1742-g89fb622b-1bionic [99.3 kB] 2021-03-09T09:32:58.031 INFO:teuthology.orchestra.run.smithi143.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 17.0.0-1742-g89fb622b-1bionic [414 kB] 2021-03-09T09:32:58.093 INFO:teuthology.orchestra.run.smithi143.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 17.0.0-1742-g89fb622b-1bionic [21.2 MB] 2021-03-09T09:32:59.873 INFO:teuthology.orchestra.run.smithi173.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-1742-g89fb622b-1bionic [19.9 kB] 2021-03-09T09:32:59.873 INFO:teuthology.orchestra.run.smithi173.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-1742-g89fb622b-1bionic [10.2 MB] 2021-03-09T09:33:00.304 INFO:teuthology.orchestra.run.smithi173.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-1742-g89fb622b-1bionic [77.7 kB] 2021-03-09T09:33:00.691 INFO:teuthology.orchestra.run.smithi143.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 17.0.0-1742-g89fb622b-1bionic [5,646 kB] 2021-03-09T09:33:00.976 INFO:teuthology.orchestra.run.smithi143.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 17.0.0-1742-g89fb622b-1bionic [184 kB] 2021-03-09T09:33:00.986 INFO:teuthology.orchestra.run.smithi143.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 17.0.0-1742-g89fb622b-1bionic [1,388 kB] 2021-03-09T09:33:01.015 INFO:teuthology.orchestra.run.smithi173.stdout:Fetched 166 MB in 6s (29.2 MB/s) 2021-03-09T09:33:01.058 INFO:teuthology.orchestra.run.smithi143.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 17.0.0-1742-g89fb622b-1bionic [6,670 kB] 2021-03-09T09:33:01.159 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-09T09:33:01.197 INFO:teuthology.orchestra.run.smithi173.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-09T09:33:01.200 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-09T09:33:01.241 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T09:33:01.398 INFO:teuthology.orchestra.run.smithi143.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 17.0.0-1742-g89fb622b-1bionic [24.9 MB] 2021-03-09T09:33:01.469 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libjaeger. 2021-03-09T09:33:01.479 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../001-libjaeger_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:01.518 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:01.863 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-09T09:33:01.874 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-09T09:33:01.963 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T09:33:02.252 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../003-librbd1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:02.350 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librbd1 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T09:33:02.655 INFO:teuthology.orchestra.run.smithi143.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 17.0.0-1742-g89fb622b-1bionic [3,812 B] 2021-03-09T09:33:02.655 INFO:teuthology.orchestra.run.smithi143.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 17.0.0-1742-g89fb622b-1bionic [771 kB] 2021-03-09T09:33:02.686 INFO:teuthology.orchestra.run.smithi143.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 17.0.0-1742-g89fb622b-1bionic [2,153 kB] 2021-03-09T09:33:02.796 INFO:teuthology.orchestra.run.smithi143.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 17.0.0-1742-g89fb622b-1bionic [63.1 kB] 2021-03-09T09:33:02.798 INFO:teuthology.orchestra.run.smithi143.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 17.0.0-1742-g89fb622b-1bionic [68.9 kB] 2021-03-09T09:33:02.802 INFO:teuthology.orchestra.run.smithi143.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 17.0.0-1742-g89fb622b-1bionic [2,333 kB] 2021-03-09T09:33:02.921 INFO:teuthology.orchestra.run.smithi143.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 17.0.0-1742-g89fb622b-1bionic [7,881 kB] 2021-03-09T09:33:02.999 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../004-librados2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:03.121 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librados2 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T09:33:03.317 INFO:teuthology.orchestra.run.smithi143.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 17.0.0-1742-g89fb622b-1bionic [3,928 B] 2021-03-09T09:33:03.317 INFO:teuthology.orchestra.run.smithi143.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 17.0.0-1742-g89fb622b-1bionic [22.5 kB] 2021-03-09T09:33:03.318 INFO:teuthology.orchestra.run.smithi143.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 17.0.0-1742-g89fb622b-1bionic [45.2 MB] 2021-03-09T09:33:03.749 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libcephfs2. 2021-03-09T09:33:03.761 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:03.799 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:04.110 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rados. 2021-03-09T09:33:04.121 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../006-python3-rados_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:04.160 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:04.504 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-09T09:33:04.515 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:04.554 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:04.848 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cephfs. 2021-03-09T09:33:04.859 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:04.898 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:05.192 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-09T09:33:05.203 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:05.234 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:05.468 INFO:teuthology.orchestra.run.smithi143.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 17.0.0-1742-g89fb622b-1bionic [19.9 kB] 2021-03-09T09:33:05.469 INFO:teuthology.orchestra.run.smithi143.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 17.0.0-1742-g89fb622b-1bionic [10.2 MB] 2021-03-09T09:33:05.503 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-prettytable. 2021-03-09T09:33:05.515 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-03-09T09:33:05.553 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-09T09:33:05.831 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rbd. 2021-03-09T09:33:05.842 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:05.881 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:05.920 INFO:teuthology.orchestra.run.smithi143.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 17.0.0-1742-g89fb622b-1bionic [77.7 kB] 2021-03-09T09:33:06.383 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-09T09:33:06.395 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-09T09:33:06.459 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T09:33:06.621 INFO:teuthology.orchestra.run.smithi143.stdout:Fetched 166 MB in 11s (14.9 MB/s) 2021-03-09T09:33:06.761 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-09T09:33:06.769 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-09T09:33:06.773 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-09T09:33:06.803 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T09:33:06.804 INFO:teuthology.orchestra.run.smithi143.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-09T09:33:06.806 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-09T09:33:06.851 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T09:33:07.078 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjaeger. 2021-03-09T09:33:07.090 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../001-libjaeger_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:07.120 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:07.122 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package librgw2. 2021-03-09T09:33:07.134 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../014-librgw2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:07.197 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:07.422 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-09T09:33:07.434 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-09T09:33:07.514 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T09:33:07.758 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rgw. 2021-03-09T09:33:07.768 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:07.808 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:07.855 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../003-librbd1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:07.977 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librbd1 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T09:33:08.152 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-09T09:33:08.163 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-09T09:33:08.202 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T09:33:08.463 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-09T09:33:08.475 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-09T09:33:08.505 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T09:33:08.617 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../004-librados2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:08.740 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librados2 (17.0.0-1742-g89fb622b-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-09T09:33:08.874 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package liboath0. 2021-03-09T09:33:08.887 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-03-09T09:33:08.924 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-09T09:33:09.235 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libradosstriper1. 2021-03-09T09:33:09.246 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:09.284 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:09.410 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs2. 2021-03-09T09:33:09.422 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../005-libcephfs2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:09.460 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:09.704 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-common. 2021-03-09T09:33:09.717 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../020-ceph-common_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:09.754 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:10.210 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rados. 2021-03-09T09:33:10.210 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../006-python3-rados_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:10.210 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:10.211 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-09T09:33:10.211 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../007-python3-ceph-argparse_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:10.211 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:10.459 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cephfs. 2021-03-09T09:33:10.471 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../008-python3-cephfs_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:10.509 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:10.837 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-09T09:33:10.848 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../009-python3-ceph-common_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:10.887 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:11.348 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-prettytable. 2021-03-09T09:33:11.359 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-03-09T09:33:11.398 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-09T09:33:11.651 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rbd. 2021-03-09T09:33:11.663 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../011-python3-rbd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:11.701 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:11.709 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-base. 2021-03-09T09:33:11.721 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../021-ceph-base_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:11.768 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:12.029 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-09T09:33:12.041 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-09T09:33:12.079 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T09:33:12.407 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-09T09:33:12.418 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-09T09:33:12.457 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T09:33:12.748 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-dateutil. 2021-03-09T09:33:12.760 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-03-09T09:33:12.776 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package librgw2. 2021-03-09T09:33:12.789 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../014-librgw2_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:12.797 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-09T09:33:12.826 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:13.108 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-09T09:33:13.120 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-09T09:33:13.158 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-09T09:33:13.379 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rgw. 2021-03-09T09:33:13.390 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../015-python3-rgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:13.420 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:13.436 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-09T09:33:13.447 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-09T09:33:13.486 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-09T09:33:13.748 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-09T09:33:13.761 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-09T09:33:13.798 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T09:33:13.822 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cryptography. 2021-03-09T09:33:13.834 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-09T09:33:13.872 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T09:33:14.101 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-09T09:33:14.113 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-09T09:33:14.151 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T09:33:14.233 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-openssl. 2021-03-09T09:33:14.245 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-09T09:33:14.309 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T09:33:14.529 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package liboath0. 2021-03-09T09:33:14.536 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-09T09:33:14.540 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-03-09T09:33:14.547 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:14.570 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-09T09:33:14.577 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:14.882 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libradosstriper1. 2021-03-09T09:33:14.894 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../019-libradosstriper1_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:14.931 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:14.940 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-09T09:33:14.951 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-09T09:33:14.990 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-09T09:33:15.318 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-09T09:33:15.331 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-09T09:33:15.368 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-09T09:33:15.385 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-common. 2021-03-09T09:33:15.396 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../020-ceph-common_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:15.435 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:15.704 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jwt. 2021-03-09T09:33:15.717 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-09T09:33:15.754 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-09T09:33:16.066 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-logutils. 2021-03-09T09:33:16.078 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-03-09T09:33:16.132 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-09T09:33:16.435 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-09T09:33:16.446 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-09T09:33:16.485 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-09T09:33:16.804 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-mako. 2021-03-09T09:33:16.816 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-09T09:33:16.854 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-09T09:33:17.190 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-09T09:33:17.202 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-09T09:33:17.240 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-09T09:33:17.357 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-base. 2021-03-09T09:33:17.368 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../021-ceph-base_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:17.416 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:17.534 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-09T09:33:17.547 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-09T09:33:17.576 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-09T09:33:17.870 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-webob. 2021-03-09T09:33:17.882 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-09T09:33:17.920 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T09:33:18.265 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-bs4. 2021-03-09T09:33:18.276 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-03-09T09:33:18.315 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-09T09:33:18.337 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-dateutil. 2021-03-09T09:33:18.350 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-03-09T09:33:18.387 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-09T09:33:18.634 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-waitress. 2021-03-09T09:33:18.647 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-03-09T09:33:18.684 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-09T09:33:18.731 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-09T09:33:18.744 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-09T09:33:18.781 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-09T09:33:19.023 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-tempita. 2021-03-09T09:33:19.036 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-03-09T09:33:19.051 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-09T09:33:19.063 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-09T09:33:19.078 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-09T09:33:19.101 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-09T09:33:19.378 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cryptography. 2021-03-09T09:33:19.391 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-09T09:33:19.406 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-paste. 2021-03-09T09:33:19.418 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-09T09:33:19.428 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T09:33:19.456 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T09:33:19.767 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-09T09:33:19.778 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-09T09:33:19.807 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-openssl. 2021-03-09T09:33:19.818 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-09T09:33:19.842 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T09:33:19.856 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T09:33:20.143 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-09T09:33:20.144 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-09T09:33:20.155 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:20.157 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-09T09:33:20.193 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:20.194 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-09T09:33:20.497 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-webtest. 2021-03-09T09:33:20.509 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-09T09:33:20.547 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-09T09:33:20.556 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T09:33:20.559 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-09T09:33:20.597 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-09T09:33:20.842 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pecan. 2021-03-09T09:33:20.852 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-03-09T09:33:20.883 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-09T09:33:20.909 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-09T09:33:20.920 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-09T09:33:20.959 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-09T09:33:21.188 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-09T09:33:21.201 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-09T09:33:21.238 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T09:33:21.329 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jwt. 2021-03-09T09:33:21.340 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../030-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-09T09:33:21.379 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-09T09:33:21.648 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr. 2021-03-09T09:33:21.660 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:21.690 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-logutils. 2021-03-09T09:33:21.697 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:21.701 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../031-python3-logutils_0.3.3-5_all.deb ... 2021-03-09T09:33:21.740 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-09T09:33:21.993 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-09T09:33:22.005 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../032-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-09T09:33:22.034 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-09T09:33:22.176 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mon. 2021-03-09T09:33:22.189 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:22.226 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:22.329 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-mako. 2021-03-09T09:33:22.341 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../033-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-09T09:33:22.379 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-09T09:33:22.690 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-09T09:33:22.703 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../034-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-09T09:33:22.740 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-09T09:33:23.043 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-09T09:33:23.055 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../035-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-09T09:33:23.093 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-09T09:33:23.213 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-osd. 2021-03-09T09:33:23.224 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:23.263 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:23.370 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webob. 2021-03-09T09:33:23.382 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../036-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-09T09:33:23.420 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T09:33:23.740 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-bs4. 2021-03-09T09:33:23.753 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../037-python3-bs4_4.6.0-1_all.deb ... 2021-03-09T09:33:23.782 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-09T09:33:24.101 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-waitress. 2021-03-09T09:33:24.114 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../038-python3-waitress_1.0.1-1_all.deb ... 2021-03-09T09:33:24.151 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-09T09:33:24.488 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-tempita. 2021-03-09T09:33:24.499 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-2_all.deb ... 2021-03-09T09:33:24.538 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-09T09:33:24.824 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-paste. 2021-03-09T09:33:24.835 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../040-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-09T09:33:24.873 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T09:33:25.204 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-09T09:33:25.214 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-09T09:33:25.253 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T09:33:25.310 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph. 2021-03-09T09:33:25.322 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../049-ceph_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:25.360 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:25.556 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-09T09:33:25.567 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../042-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-09T09:33:25.606 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-09T09:33:25.680 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-fuse. 2021-03-09T09:33:25.691 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:25.729 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:25.909 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webtest. 2021-03-09T09:33:25.921 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../043-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-09T09:33:25.951 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T09:33:26.171 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mds. 2021-03-09T09:33:26.183 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:26.221 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:26.329 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pecan. 2021-03-09T09:33:26.340 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../044-python3-pecan_1.2.1-2_all.deb ... 2021-03-09T09:33:26.378 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-09T09:33:27.425 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-09T09:33:27.438 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../045-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-09T09:33:27.465 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package cephadm. 2021-03-09T09:33:27.475 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T09:33:27.478 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../052-cephadm_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:27.515 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:27.826 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr. 2021-03-09T09:33:27.838 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../046-ceph-mgr_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:27.867 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:27.876 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jinja2. 2021-03-09T09:33:27.889 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-09T09:33:27.927 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T09:33:28.271 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-09T09:33:28.283 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:28.321 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:28.328 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mon. 2021-03-09T09:33:28.341 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../047-ceph-mon_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:28.379 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:28.657 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-09T09:33:28.669 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-03-09T09:33:28.707 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-09T09:33:29.002 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-routes. 2021-03-09T09:33:29.015 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-03-09T09:33:29.052 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-09T09:33:29.329 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-09T09:33:29.340 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-osd. 2021-03-09T09:33:29.342 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:29.351 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../048-ceph-osd_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:29.379 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:29.390 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:30.312 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-numpy. 2021-03-09T09:33:30.326 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-09T09:33:30.370 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T09:33:30.935 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-decorator. 2021-03-09T09:33:30.948 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-09T09:33:30.993 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-09T09:33:31.262 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-scipy. 2021-03-09T09:33:31.274 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-09T09:33:31.312 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T09:33:31.428 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph. 2021-03-09T09:33:31.440 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../049-ceph_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:31.478 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:31.805 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-fuse. 2021-03-09T09:33:31.818 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../050-ceph-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:31.855 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:32.267 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mds. 2021-03-09T09:33:32.278 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../051-ceph-mds_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:32.316 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:32.645 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-09T09:33:32.659 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-09T09:33:32.703 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-09T09:33:32.817 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package cephadm. 2021-03-09T09:33:32.829 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../052-cephadm_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:32.867 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:33.165 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-joblib. 2021-03-09T09:33:33.170 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jinja2. 2021-03-09T09:33:33.178 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-09T09:33:33.183 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../053-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-09T09:33:33.215 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-09T09:33:33.220 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T09:33:33.532 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-09T09:33:33.545 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../054-ceph-mgr-cephadm_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:33.562 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-sklearn. 2021-03-09T09:33:33.577 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-09T09:33:33.581 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:33.621 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-09T09:33:33.893 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-09T09:33:33.906 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../055-python3-repoze.lru_0.7-1_all.deb ... 2021-03-09T09:33:33.968 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-09T09:33:34.161 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-09T09:33:34.173 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:34.210 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:34.271 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-routes. 2021-03-09T09:33:34.283 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../056-python3-routes_2.4.1-1_all.deb ... 2021-03-09T09:33:34.321 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-09T09:33:34.633 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-09T09:33:34.645 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:34.683 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-monotonic. 2021-03-09T09:33:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-09T09:33:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-09T09:33:35.618 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-numpy. 2021-03-09T09:33:35.631 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-09T09:33:35.659 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T09:33:35.691 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-fasteners. 2021-03-09T09:33:35.703 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-09T09:33:35.741 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-09T09:33:36.086 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-httplib2. 2021-03-09T09:33:36.099 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-09T09:33:36.144 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T09:33:36.283 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-decorator. 2021-03-09T09:33:36.301 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-09T09:33:36.333 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-09T09:33:36.463 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-09T09:33:36.475 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-09T09:33:36.513 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-09T09:33:36.628 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-scipy. 2021-03-09T09:33:36.639 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-09T09:33:36.678 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T09:33:36.842 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-09T09:33:36.855 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-09T09:33:36.900 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T09:33:37.219 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-rsa. 2021-03-09T09:33:37.233 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-09T09:33:37.269 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-09T09:33:37.581 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-09T09:33:37.593 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-09T09:33:37.630 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T09:33:37.837 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-09T09:33:37.851 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-09T09:33:37.895 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-09T09:33:38.102 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-09T09:33:38.114 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-09T09:33:38.151 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-09T09:33:38.324 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-joblib. 2021-03-09T09:33:38.337 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-09T09:33:38.373 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-09T09:33:38.496 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-setuptools. 2021-03-09T09:33:38.508 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-09T09:33:38.546 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-09T09:33:38.702 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sklearn. 2021-03-09T09:33:38.715 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-09T09:33:38.752 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-09T09:33:38.916 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-websocket. 2021-03-09T09:33:38.929 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-09T09:33:38.974 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T09:33:39.286 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-09T09:33:39.298 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-09T09:33:39.335 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T09:33:39.344 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-09T09:33:39.355 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:39.394 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:39.773 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-09T09:33:39.785 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:39.822 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:40.142 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-09T09:33:40.154 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-09T09:33:40.183 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-09T09:33:40.522 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-monotonic. 2021-03-09T09:33:40.534 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-09T09:33:40.553 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-09T09:33:40.565 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-09T09:33:40.572 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-09T09:33:40.606 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T09:33:40.825 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-fasteners. 2021-03-09T09:33:40.838 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-09T09:33:40.883 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-09T09:33:40.889 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-09T09:33:40.903 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:40.947 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:41.186 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-httplib2. 2021-03-09T09:33:41.200 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-09T09:33:41.245 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T09:33:41.292 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-09T09:33:41.305 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-09T09:33:41.427 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-09T09:33:41.548 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-09T09:33:41.559 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-09T09:33:41.597 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-09T09:33:41.789 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-09T09:33:41.802 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-09T09:33:41.847 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T09:33:42.108 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package jq. 2021-03-09T09:33:42.122 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-09T09:33:42.166 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-09T09:33:42.168 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-09T09:33:42.180 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-09T09:33:42.218 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T09:33:42.419 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package socat. 2021-03-09T09:33:42.432 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-09T09:33:42.477 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-09T09:33:42.546 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-rsa. 2021-03-09T09:33:42.559 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-09T09:33:42.587 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-09T09:33:42.822 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-09T09:33:42.836 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-09T09:33:42.866 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-09T09:33:42.880 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-09T09:33:42.881 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T09:33:42.924 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T09:33:43.150 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package xmlstarlet. 2021-03-09T09:33:43.163 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-09T09:33:43.200 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-09T09:33:43.354 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-09T09:33:43.368 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-09T09:33:43.421 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-09T09:33:43.596 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package ceph-test. 2021-03-09T09:33:43.608 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../086-ceph-test_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:43.646 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:43.757 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-setuptools. 2021-03-09T09:33:43.770 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-09T09:33:43.799 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-09T09:33:44.419 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-websocket. 2021-03-09T09:33:44.432 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-09T09:33:44.477 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T09:33:44.773 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-09T09:33:44.786 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-09T09:33:44.814 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T09:33:45.227 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-09T09:33:45.238 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:45.277 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:45.580 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-09T09:33:45.594 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-09T09:33:45.638 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-09T09:33:45.991 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-09T09:33:46.003 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-09T09:33:46.036 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T09:33:46.353 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-09T09:33:46.366 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../079-ceph-mgr-rook_17.0.0-1742-g89fb622b-1bionic_all.deb ... 2021-03-09T09:33:46.403 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:46.764 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-09T09:33:46.778 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-09T09:33:46.900 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-09T09:33:47.024 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-09T09:33:47.037 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:47.074 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:47.236 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-09T09:33:47.248 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-09T09:33:47.286 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T09:33:47.368 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package nvme-cli. 2021-03-09T09:33:47.381 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-09T09:33:47.439 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T09:33:47.556 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package jq. 2021-03-09T09:33:47.569 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-09T09:33:47.614 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-09T09:33:47.781 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-attr. 2021-03-09T09:33:47.794 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-09T09:33:47.831 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-09T09:33:47.867 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package socat. 2021-03-09T09:33:47.879 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-09T09:33:47.917 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-09T09:33:48.126 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-funcsigs. 2021-03-09T09:33:48.143 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-09T09:33:48.184 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-09T09:33:48.246 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-09T09:33:48.260 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-09T09:33:48.304 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T09:33:48.495 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-pluggy. 2021-03-09T09:33:48.508 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-09T09:33:48.553 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-09T09:33:48.557 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package xmlstarlet. 2021-03-09T09:33:48.568 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-09T09:33:48.607 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-09T09:33:48.814 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-py. 2021-03-09T09:33:48.826 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-09T09:33:48.864 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-09T09:33:48.953 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package ceph-test. 2021-03-09T09:33:48.966 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../086-ceph-test_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:49.003 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:49.201 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python-pytest. 2021-03-09T09:33:49.214 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-09T09:33:49.250 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-09T09:33:49.587 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-attr. 2021-03-09T09:33:49.599 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-09T09:33:49.637 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-09T09:33:49.931 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-cycler. 2021-03-09T09:33:49.945 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-09T09:33:49.990 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-09T09:33:50.301 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-webencodings. 2021-03-09T09:33:50.314 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-09T09:33:50.351 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-09T09:33:50.679 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-html5lib. 2021-03-09T09:33:50.691 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-09T09:33:50.728 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-09T09:33:51.073 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-09T09:33:51.086 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-09T09:33:51.123 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T09:33:51.577 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-09T09:33:51.589 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-09T09:33:51.626 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T09:33:51.937 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-tz. 2021-03-09T09:33:51.948 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-09T09:33:51.987 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-09T09:33:52.282 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-09T09:33:52.294 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-09T09:33:52.331 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T09:33:52.382 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-09T09:33:52.396 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../087-libcephfs-dev_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:52.440 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:52.727 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package nvme-cli. 2021-03-09T09:33:52.739 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-09T09:33:52.777 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T09:33:52.971 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-olefile. 2021-03-09T09:33:52.984 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-09T09:33:53.029 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-09T09:33:53.057 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-attr. 2021-03-09T09:33:53.070 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-09T09:33:53.098 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-09T09:33:53.348 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pastescript. 2021-03-09T09:33:53.361 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-09T09:33:53.376 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-funcsigs. 2021-03-09T09:33:53.389 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-09T09:33:53.398 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-09T09:33:53.426 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-09T09:33:53.737 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-pluggy. 2021-03-09T09:33:53.752 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-09T09:33:53.793 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-09T09:33:53.796 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-09T09:33:53.806 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-03-09T09:33:53.843 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T09:33:54.057 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-py. 2021-03-09T09:33:54.070 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-09T09:33:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-09T09:33:54.213 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pluggy. 2021-03-09T09:33:54.226 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-09T09:33:54.263 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-09T09:33:54.411 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python-pytest. 2021-03-09T09:33:54.424 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-09T09:33:54.469 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-09T09:33:54.582 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-py. 2021-03-09T09:33:54.595 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-09T09:33:54.632 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-09T09:33:54.814 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-attr. 2021-03-09T09:33:54.826 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-09T09:33:54.864 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-09T09:33:54.927 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-09T09:33:54.941 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-09T09:33:54.977 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-09T09:33:55.167 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-cycler. 2021-03-09T09:33:55.180 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-09T09:33:55.217 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-09T09:33:55.280 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-pytest. 2021-03-09T09:33:55.293 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-09T09:33:55.346 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-09T09:33:55.520 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-webencodings. 2021-03-09T09:33:55.531 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-09T09:33:55.570 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-09T09:33:55.650 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-simplejson. 2021-03-09T09:33:55.664 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-09T09:33:55.707 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-09T09:33:55.873 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-html5lib. 2021-03-09T09:33:55.887 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-09T09:33:55.931 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-09T09:33:56.027 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-09T09:33:56.041 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-09T09:33:56.094 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T09:33:56.277 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-09T09:33:56.288 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-09T09:33:56.326 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T09:33:56.548 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package radosgw. 2021-03-09T09:33:56.561 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../111-radosgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:56.606 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:56.772 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-09T09:33:56.785 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-09T09:33:56.830 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T09:33:57.150 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-tz. 2021-03-09T09:33:57.162 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-09T09:33:57.200 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-09T09:33:57.511 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-09T09:33:57.523 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-09T09:33:57.561 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T09:33:57.668 INFO:teuthology.orchestra.run.smithi173.stdout:Selecting previously unselected package rbd-fuse. 2021-03-09T09:33:57.681 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:33:57.717 INFO:teuthology.orchestra.run.smithi173.stdout:Unpacking rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:33:58.011 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-09T09:33:58.184 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-olefile. 2021-03-09T09:33:58.195 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-09T09:33:58.233 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-09T09:33:58.381 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T09:33:58.545 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pastescript. 2021-03-09T09:33:58.558 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-09T09:33:58.595 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-09T09:33:58.774 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T09:33:58.899 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T09:33:58.966 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-09T09:33:58.978 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-03-09T09:33:59.015 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T09:33:59.386 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pluggy. 2021-03-09T09:33:59.398 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-09T09:33:59.435 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-09T09:33:59.519 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T09:33:59.738 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-py. 2021-03-09T09:33:59.752 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-09T09:33:59.813 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-09T09:33:59.903 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-09T09:34:00.142 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-09T09:34:00.154 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-09T09:34:00.192 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-09T09:34:00.204 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-09T09:34:00.478 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-pytest. 2021-03-09T09:34:00.489 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-09T09:34:00.497 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-09T09:34:00.528 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-09T09:34:00.699 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-09T09:34:00.841 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-09T09:34:00.848 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-simplejson. 2021-03-09T09:34:00.862 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-09T09:34:00.907 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-09T09:34:01.187 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-09T09:34:01.218 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-09T09:34:01.232 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-09T09:34:01.276 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T09:34:01.563 INFO:teuthology.orchestra.run.smithi173.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-09T09:34:01.731 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package radosgw. 2021-03-09T09:34:01.743 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../111-radosgw_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:34:01.780 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:01.856 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-09T09:34:02.158 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:02.759 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-09T09:34:02.801 INFO:teuthology.orchestra.run.smithi143.stdout:Selecting previously unselected package rbd-fuse. 2021-03-09T09:34:02.812 INFO:teuthology.orchestra.run.smithi143.stdout:Preparing to unpack .../112-rbd-fuse_17.0.0-1742-g89fb622b-1bionic_amd64.deb ... 2021-03-09T09:34:02.842 INFO:teuthology.orchestra.run.smithi143.stdout:Unpacking rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:03.052 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-09T09:34:03.136 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-09T09:34:03.178 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T09:34:03.456 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-09T09:34:03.824 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-09T09:34:03.925 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-09T09:34:04.148 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T09:34:04.466 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-09T09:34:04.520 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-09T09:34:04.742 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-py (1.5.2-1) ... 2021-03-09T09:34:04.930 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-09T09:34:04.985 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-09T09:34:05.214 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-09T09:34:05.328 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-09T09:34:05.418 INFO:teuthology.orchestra.run.smithi173.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-09T09:34:05.507 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-09T09:34:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-09T09:34:05.717 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-09T09:34:05.806 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-09T09:34:05.826 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-09T09:34:06.007 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-09T09:34:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-09T09:34:06.308 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T09:34:06.593 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-09T09:34:06.643 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-09T09:34:06.890 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-09T09:34:06.927 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-09T09:34:07.191 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-argparse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:07.304 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-09T09:34:07.508 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-09T09:34:07.630 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-09T09:34:07.793 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-09T09:34:07.919 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-09T09:34:07.957 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:08.582 INFO:teuthology.orchestra.run.smithi173.stdout:Adding system user cephadm....done 2021-03-09T09:34:08.789 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-09T09:34:08.814 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-09T09:34:09.098 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T09:34:09.140 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-09T09:34:09.224 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T09:34:09.367 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-09T09:34:09.416 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-py (1.5.2-1) ... 2021-03-09T09:34:09.618 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-09T09:34:09.735 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-09T09:34:09.961 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-09T09:34:10.012 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T09:34:10.076 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-09T09:34:10.138 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-09T09:34:10.312 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-09T09:34:10.447 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-09T09:34:10.498 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:10.640 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-09T09:34:10.816 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-09T09:34:10.933 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-09T09:34:11.117 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T09:34:11.242 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-09T09:34:11.243 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T09:34:11.518 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-09T09:34:11.594 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-09T09:34:11.886 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-09T09:34:11.938 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T09:34:12.027 INFO:teuthology.orchestra.run.smithi173.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-09T09:34:12.060 INFO:teuthology.orchestra.run.smithi173.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-09T09:34:12.213 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-09T09:34:12.281 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-09T09:34:12.514 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:12.574 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T09:34:12.892 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T09:34:13.235 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-09T09:34:13.515 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user cephadm....done 2021-03-09T09:34:13.536 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-09T09:34:13.671 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-09T09:34:13.729 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:13.855 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T09:34:13.973 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-09T09:34:14.099 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-09T09:34:14.242 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-09T09:34:14.257 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T09:34:14.347 INFO:teuthology.orchestra.run.smithi173.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-09T09:34:14.576 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-09T09:34:14.601 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-09T09:34:14.777 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T09:34:14.861 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-09T09:34:14.878 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T09:34:14.987 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-09T09:34:15.163 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T09:34:15.289 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-09T09:34:15.314 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:15.623 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-09T09:34:15.691 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T09:34:15.842 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-09T09:34:15.925 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-09T09:34:16.050 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-09T09:34:16.126 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-09T09:34:16.385 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-09T09:34:16.428 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T09:34:16.721 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-09T09:34:16.728 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-09T09:34:16.818 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-09T09:34:16.850 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-09T09:34:17.014 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-09T09:34:17.089 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-09T09:34:17.390 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-09T09:34:17.391 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-09T09:34:17.691 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-09T09:34:17.699 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-09T09:34:18.001 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-09T09:34:18.059 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-09T09:34:18.319 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T09:34:18.344 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-09T09:34:18.545 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libjaeger (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:18.662 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-09T09:34:18.671 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-09T09:34:18.955 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-09T09:34:19.058 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-09T09:34:19.081 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T09:34:19.148 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-09T09:34:19.401 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-09T09:34:19.560 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-09T09:34:19.686 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-09T09:34:19.921 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-09T09:34:20.055 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-09T09:34:20.245 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T09:34:20.457 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-09T09:34:20.580 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-09T09:34:20.582 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-09T09:34:20.867 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-09T09:34:20.906 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-09T09:34:21.032 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-09T09:34:21.160 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-09T09:34:21.392 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-09T09:34:21.427 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-09T09:34:21.652 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T09:34:21.720 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-09T09:34:21.761 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-09T09:34:21.871 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librados2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:22.030 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-09T09:34:22.088 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-09T09:34:22.340 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:22.405 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-09T09:34:22.466 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:22.592 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:22.715 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-09T09:34:22.718 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T09:34:23.049 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-09T09:34:23.069 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-09T09:34:23.346 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-09T09:34:23.426 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-09T09:34:23.727 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-09T09:34:23.765 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T09:34:23.836 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-09T09:34:24.611 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-09T09:34:24.999 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-09T09:34:25.317 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-09T09:34:25.416 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T09:34:25.652 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-09T09:34:25.717 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:25.752 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-09T09:34:25.843 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:25.994 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:26.112 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-09T09:34:26.112 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:26.305 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T09:34:26.305 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T09:34:26.371 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-09T09:34:26.497 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-09T09:34:26.580 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up librbd1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:26.632 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librados2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:26.723 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-09T09:34:26.758 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-09T09:34:26.821 INFO:teuthology.orchestra.run.smithi173.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-09T09:34:26.845 INFO:teuthology.orchestra.run.smithi173.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-09T09:34:27.067 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:27.160 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T09:34:27.193 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libcephfs-dev (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:27.302 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rados (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:27.411 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-09T09:34:27.712 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-09T09:34:27.838 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:27.972 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-09T09:34:28.013 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-09T09:34:28.290 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:28.416 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:28.417 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-09T09:34:28.542 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:28.668 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:29.203 INFO:teuthology.orchestra.run.smithi173.stdout:Adding group ceph....done 2021-03-09T09:34:29.252 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-09T09:34:29.606 INFO:teuthology.orchestra.run.smithi173.stdout:Adding system user ceph....done 2021-03-09T09:34:29.816 INFO:teuthology.orchestra.run.smithi173.stdout:Setting system user ceph properties....done 2021-03-09T09:34:29.971 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-09T09:34:30.064 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-09T09:34:30.186 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-09T09:34:30.365 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up libradosstriper1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:30.471 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:30.491 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-cephfs (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:30.634 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librgw2 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:30.756 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T09:34:30.756 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T09:34:30.760 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:30.961 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T09:34:30.961 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-09T09:34:31.148 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:31.236 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up librbd1 (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:31.274 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:31.362 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-09T09:34:31.452 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-09T09:34:31.493 INFO:teuthology.orchestra.run.smithi143.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-09T09:34:31.627 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-09T09:34:31.815 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-09T09:34:31.909 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:32.123 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T09:34:32.124 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T09:34:32.419 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:32.476 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-modules-core (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:32.554 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:32.602 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-09T09:34:32.903 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:32.973 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T09:34:32.974 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T09:34:33.029 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up rbd-fuse (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:33.188 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up python3-rbd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:33.314 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-common (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:33.566 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:33.708 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:33.857 INFO:teuthology.orchestra.run.smithi143.stdout:Adding group ceph....done 2021-03-09T09:34:33.876 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:34.099 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T09:34:34.100 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T09:34:34.234 INFO:teuthology.orchestra.run.smithi143.stdout:Adding system user ceph....done 2021-03-09T09:34:34.378 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:34.426 INFO:teuthology.orchestra.run.smithi143.stdout:Setting system user ceph properties....done 2021-03-09T09:34:34.581 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-09T09:34:34.595 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T09:34:34.595 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T09:34:34.814 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-09T09:34:34.904 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:35.022 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:35.115 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up radosgw (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:35.165 INFO:teuthology.orchestra.run.smithi173.stdout:Setting up ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-09T09:34:35.384 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T09:34:35.385 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-09T09:34:35.526 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-09T09:34:35.767 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-test (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:35.918 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-base (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:36.246 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-09T09:34:36.519 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:36.726 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T09:34:36.727 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-09T09:34:36.987 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:37.130 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-osd (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:37.568 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T09:34:37.568 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-09T09:34:37.762 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-09T09:34:37.805 INFO:teuthology.orchestra.run.smithi173.stdout:ureadahead will be reprofiled on next reboot 2021-03-09T09:34:37.823 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-cephadm (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:37.854 INFO:teuthology.orchestra.run.smithi173.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-09T09:34:37.966 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-rook (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:38.109 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mds (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:38.320 INFO:teuthology.orchestra.run.smithi173.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-09T09:34:38.323 DEBUG:teuthology.parallel:result is None 2021-03-09T09:34:38.559 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T09:34:38.559 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-09T09:34:38.844 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mon (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:39.063 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T09:34:39.063 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-09T09:34:39.337 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-k8sevents (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:39.463 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph-mgr-dashboard (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:39.605 INFO:teuthology.orchestra.run.smithi143.stdout:Setting up ceph (17.0.0-1742-g89fb622b-1bionic) ... 2021-03-09T09:34:39.732 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-09T09:34:39.925 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-09T09:34:41.952 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-09T09:34:41.986 INFO:teuthology.orchestra.run.smithi143.stdout:ureadahead will be reprofiled on next reboot 2021-03-09T09:34:42.035 INFO:teuthology.orchestra.run.smithi143.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-09T09:34:42.473 INFO:teuthology.orchestra.run.smithi143.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-09T09:34:42.475 DEBUG:teuthology.parallel:result is None 2021-03-09T09:34:42.478 DEBUG:teuthology.orchestra.run.smithi143:> sudo lsb_release -is 2021-03-09T09:34:42.533 INFO:teuthology.orchestra.run.smithi143.stdout:Ubuntu 2021-03-09T09:34:42.538 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:34:42.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:34:42.692 DEBUG:teuthology.orchestra.run.smithi143:> dpkg-query -W -f '${Version}' ceph 2021-03-09T09:34:42.716 INFO:teuthology.orchestra.run.smithi143.stdout:17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:34:42.716 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:34:42.717 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742-g89fb622b-1bionic is installed. 2021-03-09T09:34:42.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo lsb_release -is 2021-03-09T09:34:42.772 INFO:teuthology.orchestra.run.smithi173.stdout:Ubuntu 2021-03-09T09:34:42.777 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-09T09:34:42.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T09:34:42.924 DEBUG:teuthology.orchestra.run.smithi173:> dpkg-query -W -f '${Version}' ceph 2021-03-09T09:34:42.948 INFO:teuthology.orchestra.run.smithi173.stdout:17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:34:42.949 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742-g89fb622b-1bionic 2021-03-09T09:34:42.949 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742-g89fb622b-1bionic is installed. 2021-03-09T09:34:42.949 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-09T09:34:42.955 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:34:42.956 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T09:34:42.971 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:34:42.971 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T09:34:43.004 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-09T09:34:43.004 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:34:43.005 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T09:34:43.032 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T09:34:43.092 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:34:43.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T09:34:43.107 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T09:34:43.163 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-09T09:34:43.164 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:34:43.165 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T09:34:43.180 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T09:34:43.236 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:34:43.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T09:34:43.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T09:34:43.307 INFO:teuthology.run_tasks:Running task ceph... 2021-03-09T09:34:43.386 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-03-09T09:34:43.387 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /var/log/ceph 2021-03-09T09:34:43.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2021-03-09T09:34:43.429 INFO:tasks.ceph:Disabling ceph logrotate... 2021-03-09T09:34:43.430 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-09T09:34:43.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-09T09:34:43.490 INFO:tasks.ceph:Creating extra log directories... 2021-03-09T09:34:43.490 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-09T09:34:43.531 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-09T09:34:43.565 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-03-09T09:34:43.565 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'slow request', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2021-03-09T09:34:43.566 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949128', 'branch': 'wip-kefu-testing-2021-03-09-1428', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/pool-create-delete}', 'email': 'tchaikov@gmail.com', 'first_in_suite': False, 'job_id': '5949128', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '18.04', 'overrides': {'admin_socket': {'branch': 'wip-kefu-testing-2021-03-09-1428'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'slow request', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'mon_bind_msgr2': False, 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-kefu-testing-2021-03-09-1428', 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}}, 'owner': 'scheduled_kchai@teuthology', 'priority': 60, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-kefu-testing-2021-03-09-1428', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'targets': {'smithi143.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYlkzOm2rvOuAmgVE+jCvsGyn0dB2+sZy0kvo8mvAZPjsSjFrbSrGBBVn3OOlfKALGntG/EG+3SGHpwX2fHqnKIaYTGNkRg6lS9f334YeZj+Md4Oa1YJt6H9wtQQr74hQEUAkIkRRBepq5aclptcLucERYMf/81Fg2m5hnx22QyS5/rjLiz/AMSaTTKSdgSYQLwKIlarQo1F/MEpFRHE4vmQAXbXGQn/iwKaiuwk6J0ojL1uW2dZ9P7tsxawZOJzQGXjzwClAsMklbV/3U4ekSPT/lDuG4bB7YxFzKulUYLlGmcvQOa5TGaGbg2mqNmrBg/mdx2Ho5ggc/9VfirBM7', 'smithi173.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOyMvWSMPEqNIM7iCReHZ79iR8+a8F4KT2ENGt0CXrU73TGJZOjKHfNG5iIUACUllwOLUrQ8+213akipy58vlWXu1JsR69kPvSGmXnl4KbeUlEw4SwYw4DuFN7m3P3eD0UtW304pB0pkKTDcp2XJ1Ew/BXjaxUViOn6/dICx8/UrgNhv9qDYg8Ulg5cLf6swndZY9uFrDxlyeCpodHO2m24fIeRAvdABWriVkEEsmoPG8ieMeVEOoLBV3vk5Y6GQ6xGbUkX/BMKcURnaMx37GMiEnpuXtCizOrcRw3nw298fx1sryeOybHu6BxYccx9DiHZWuWJbMss2dcaIFac7Wd'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'freeze_mon_duration': 20, 'freeze_mon_probability': 10, 'revive_delay': 20, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology_branch': 'master', 'teuthology_sha1': '3082387bba74fcd24c9700593d10418152d53c97', 'timestamp': '2021-03-09_08:22:40', 'tube': 'smithi', 'user': 'kchai', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.25973'} 2021-03-09T09:34:43.568 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-03-09T09:34:43.587 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-03-09T09:34:43.595 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/run/ceph 2021-03-09T09:34:43.635 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2021-03-09T09:34:43.650 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:34:43.651 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T09:34:43.695 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T09:34:43.696 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T09:34:43.744 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T09:34:43.744 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:34:43.744 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 341 Links: 1 2021-03-09T09:34:43.744 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:34:43.745 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2021-03-09 09:32:13.007796479 +0000 2021-03-09T09:34:43.745 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2021-03-09 09:32:12.915796481 +0000 2021-03-09T09:34:43.745 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2021-03-09 09:32:12.915796481 +0000 2021-03-09T09:34:43.745 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2021-03-09T09:34:43.753 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2021-03-09T09:34:43.753 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2021-03-09T09:34:43.753 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000352096 s, 1.5 MB/s 2021-03-09T09:34:43.758 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T09:34:43.807 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T09:34:43.808 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:34:43.808 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 347 Links: 1 2021-03-09T09:34:43.808 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:34:43.808 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2021-03-09 09:32:13.271796473 +0000 2021-03-09T09:34:43.809 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2021-03-09 09:32:13.183796475 +0000 2021-03-09T09:34:43.809 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2021-03-09 09:32:13.183796475 +0000 2021-03-09T09:34:43.809 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2021-03-09T09:34:43.816 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2021-03-09T09:34:43.817 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2021-03-09T09:34:43.817 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000312815 s, 1.6 MB/s 2021-03-09T09:34:43.821 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T09:34:43.872 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T09:34:43.873 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:34:43.873 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 353 Links: 1 2021-03-09T09:34:43.874 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:34:43.874 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2021-03-09 09:32:13.519796467 +0000 2021-03-09T09:34:43.875 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2021-03-09 09:32:13.415796470 +0000 2021-03-09T09:34:43.875 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2021-03-09 09:32:13.415796470 +0000 2021-03-09T09:34:43.876 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2021-03-09T09:34:43.882 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2021-03-09T09:34:43.882 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2021-03-09T09:34:43.883 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000341531 s, 1.5 MB/s 2021-03-09T09:34:43.886 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T09:34:43.936 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T09:34:43.936 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:34:43.936 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 6h/6d Inode: 360 Links: 1 2021-03-09T09:34:43.937 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:34:43.937 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2021-03-09 09:32:13.747796462 +0000 2021-03-09T09:34:43.937 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2021-03-09 09:32:13.635796465 +0000 2021-03-09T09:34:43.938 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2021-03-09 09:32:13.635796465 +0000 2021-03-09T09:34:43.938 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: - 2021-03-09T09:34:43.945 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2021-03-09T09:34:43.946 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2021-03-09T09:34:43.946 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000386901 s, 1.3 MB/s 2021-03-09T09:34:43.949 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-03-09T09:34:43.950 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:34:43.950 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T09:34:43.957 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T09:34:43.958 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T09:34:44.004 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T09:34:44.004 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:34:44.004 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-03-09T09:34:44.005 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:34:44.005 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2021-03-09 09:32:04.359693118 +0000 2021-03-09T09:34:44.005 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2021-03-09 09:32:04.239693119 +0000 2021-03-09T09:34:44.005 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2021-03-09 09:32:04.239693119 +0000 2021-03-09T09:34:44.006 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2021-03-09T09:34:44.013 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2021-03-09T09:34:44.014 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2021-03-09T09:34:44.014 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000386633 s, 1.3 MB/s 2021-03-09T09:34:44.018 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T09:34:44.068 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T09:34:44.068 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:34:44.069 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 347 Links: 1 2021-03-09T09:34:44.069 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:34:44.069 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2021-03-09 09:32:04.647693115 +0000 2021-03-09T09:34:44.069 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2021-03-09 09:32:04.519693116 +0000 2021-03-09T09:34:44.069 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2021-03-09 09:32:04.519693116 +0000 2021-03-09T09:34:44.070 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2021-03-09T09:34:44.077 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2021-03-09T09:34:44.077 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2021-03-09T09:34:44.078 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000381317 s, 1.3 MB/s 2021-03-09T09:34:44.082 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T09:34:44.132 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T09:34:44.132 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:34:44.133 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 353 Links: 1 2021-03-09T09:34:44.133 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:34:44.133 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2021-03-09 09:32:04.795693113 +0000 2021-03-09T09:34:44.133 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2021-03-09 09:32:04.791693113 +0000 2021-03-09T09:34:44.133 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2021-03-09 09:32:04.791693113 +0000 2021-03-09T09:34:44.134 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2021-03-09T09:34:44.141 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2021-03-09T09:34:44.142 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2021-03-09T09:34:44.142 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00040914 s, 1.3 MB/s 2021-03-09T09:34:44.146 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T09:34:44.196 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T09:34:44.196 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T09:34:44.196 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 359 Links: 1 2021-03-09T09:34:44.197 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T09:34:44.197 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2021-03-09 09:32:05.023693111 +0000 2021-03-09T09:34:44.197 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2021-03-09 09:32:05.019693111 +0000 2021-03-09T09:34:44.197 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2021-03-09 09:32:05.019693111 +0000 2021-03-09T09:34:44.197 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: - 2021-03-09T09:34:44.205 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2021-03-09T09:34:44.205 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2021-03-09T09:34:44.206 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000342552 s, 1.5 MB/s 2021-03-09T09:34:44.209 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2021-03-09T09:34:44.210 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2021-03-09T09:34:44.210 INFO:tasks.ceph:Generating config... 2021-03-09T09:34:44.212 INFO:tasks.ceph:[client] debug monc = 20 2021-03-09T09:34:44.213 INFO:tasks.ceph:[client] debug ms = 1 2021-03-09T09:34:44.213 INFO:tasks.ceph:[global] mon client directed command retry = 5 2021-03-09T09:34:44.213 INFO:tasks.ceph:[global] mon election default strategy = 1 2021-03-09T09:34:44.214 INFO:tasks.ceph:[global] ms bind msgr2 = False 2021-03-09T09:34:44.214 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2021-03-09T09:34:44.214 INFO:tasks.ceph:[global] ms type = async 2021-03-09T09:34:44.215 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-09T09:34:44.215 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-09T09:34:44.215 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-09T09:34:44.215 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-09T09:34:44.216 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-09T09:34:44.216 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2021-03-09T09:34:44.216 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2021-03-09T09:34:44.217 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2021-03-09T09:34:44.217 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2021-03-09T09:34:44.217 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2021-03-09T09:34:44.218 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2021-03-09T09:34:44.218 INFO:tasks.ceph:[mon] mon scrub interval = 300 2021-03-09T09:34:44.218 INFO:tasks.ceph:[mon] paxos service trim min = 5 2021-03-09T09:34:44.219 INFO:tasks.ceph:[osd] bdev async discard = True 2021-03-09T09:34:44.219 INFO:tasks.ceph:[osd] bdev enable discard = True 2021-03-09T09:34:44.219 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2021-03-09T09:34:44.219 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-03-09T09:34:44.220 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-03-09T09:34:44.220 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-03-09T09:34:44.220 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-03-09T09:34:44.221 INFO:tasks.ceph:[osd] debug filestore = 20 2021-03-09T09:34:44.221 INFO:tasks.ceph:[osd] debug journal = 20 2021-03-09T09:34:44.221 INFO:tasks.ceph:[osd] debug ms = 20 2021-03-09T09:34:44.221 INFO:tasks.ceph:[osd] debug osd = 25 2021-03-09T09:34:44.222 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-03-09T09:34:44.222 INFO:tasks.ceph:[osd] mon client ping interval = 4 2021-03-09T09:34:44.222 INFO:tasks.ceph:[osd] mon client ping timeout = 12 2021-03-09T09:34:44.223 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-03-09T09:34:44.223 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-03-09T09:34:44.223 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-03-09T09:34:44.223 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2021-03-09T09:34:44.224 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2021-03-09T09:34:44.224 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-03-09T09:34:44.224 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-03-09T09:34:44.225 INFO:tasks.ceph:[osd] osd op queue = debug_random 2021-03-09T09:34:44.225 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2021-03-09T09:34:44.225 INFO:tasks.ceph:Setting up mon.a... 2021-03-09T09:34:44.226 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-03-09T09:34:44.314 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/ceph.keyring 2021-03-09T09:34:44.320 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-03-09T09:34:44.409 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T09:34:44.424 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.143:6789]'), ('mon.b', '[v1:172.21.15.143:6790]'), ('mon.c', '[v1:172.21.15.143:6791]'), ('mon.d', '[v1:172.21.15.143:6792]'), ('mon.e', '[v1:172.21.15.143:6793]'), ('mon.f', '[v1:172.21.15.173:6789]'), ('mon.g', '[v1:172.21.15.173:6790]'), ('mon.h', '[v1:172.21.15.173:6791]'), ('mon.i', '[v1:172.21.15.173:6792]')] 2021-03-09T09:34:44.425 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.143:6789],[v1:172.21.15.143:6790],[v1:172.21.15.143:6791],[v1:172.21.15.143:6792],[v1:172.21.15.143:6793],[v1:172.21.15.173:6789],[v1:172.21.15.173:6790],[v1:172.21.15.173:6791],[v1:172.21.15.173:6792]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2021-03-09T09:34:44.426 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:34:44.427 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-03-09T09:34:44.471 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.143:6789]' --addv b '[v1:172.21.15.143:6790]' --addv c '[v1:172.21.15.143:6791]' --addv d '[v1:172.21.15.143:6792]' --addv e '[v1:172.21.15.143:6793]' --addv f '[v1:172.21.15.173:6789]' --addv g '[v1:172.21.15.173:6790]' --addv h '[v1:172.21.15.173:6791]' --addv i '[v1:172.21.15.173:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:34:44.619 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:34:44.619 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool: generated fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:34:44.620 INFO:teuthology.orchestra.run.smithi143.stdout:setting min_mon_release = octopus 2021-03-09T09:34:44.620 INFO:teuthology.orchestra.run.smithi143.stdout:epoch 0 2021-03-09T09:34:44.620 INFO:teuthology.orchestra.run.smithi143.stdout:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:34:44.621 INFO:teuthology.orchestra.run.smithi143.stdout:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:34:44.621 INFO:teuthology.orchestra.run.smithi143.stdout:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:34:44.621 INFO:teuthology.orchestra.run.smithi143.stdout:min_mon_release 15 (octopus) 2021-03-09T09:34:44.622 INFO:teuthology.orchestra.run.smithi143.stdout:election_strategy: 1 2021-03-09T09:34:44.622 INFO:teuthology.orchestra.run.smithi143.stdout:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:34:44.622 INFO:teuthology.orchestra.run.smithi143.stdout:1: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:34:44.623 INFO:teuthology.orchestra.run.smithi143.stdout:2: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:34:44.623 INFO:teuthology.orchestra.run.smithi143.stdout:3: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:34:44.623 INFO:teuthology.orchestra.run.smithi143.stdout:4: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:34:44.624 INFO:teuthology.orchestra.run.smithi143.stdout:5: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:34:44.624 INFO:teuthology.orchestra.run.smithi143.stdout:6: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:34:44.625 INFO:teuthology.orchestra.run.smithi143.stdout:7: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:34:44.625 INFO:teuthology.orchestra.run.smithi143.stdout:8: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:34:44.625 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2021-03-09T09:34:44.627 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-03-09T09:34:44.634 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8a34d6d4-5d27-4c75-9119-8e930fb2bed7... 2021-03-09T09:34:44.635 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-03-09T09:34:44.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-03-09T09:34:44.709 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2021-03-09T09:34:44.709 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2021-03-09T09:34:44.710 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-09T09:34:44.710 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2021-03-09T09:34:44.711 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.711 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2021-03-09T09:34:44.711 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.712 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2021-03-09T09:34:44.712 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.713 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2021-03-09T09:34:44.713 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2021-03-09T09:34:44.713 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.714 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2021-03-09T09:34:44.714 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2021-03-09T09:34:44.715 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.715 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-09T09:34:44.715 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2021-03-09T09:34:44.716 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.716 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2021-03-09T09:34:44.716 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-09T09:34:44.717 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.717 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2021-03-09T09:34:44.718 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.718 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2021-03-09T09:34:44.718 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2021-03-09T09:34:44.719 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2021-03-09T09:34:44.719 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2021-03-09T09:34:44.719 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2021-03-09T09:34:44.720 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2021-03-09T09:34:44.720 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2021-03-09T09:34:44.721 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-09T09:34:44.721 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-09T09:34:44.721 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2021-03-09T09:34:44.722 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.722 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-09T09:34:44.722 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.723 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2021-03-09T09:34:44.723 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.723 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2021-03-09T09:34:44.723 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.724 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2021-03-09T09:34:44.724 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2021-03-09T09:34:44.724 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2021-03-09T09:34:44.724 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = "[v1:172.21.15.143:6789],[v1:172.21.15.143:6790],[v1:172.21.15.143:6791],[v1:172.21.15.143:6792],[v1:172.21.15.143:6793],[v1:172.21.15.173:6789],[v1:172.21.15.173:6790],[v1:172.21.15.173:6791],[v1:172.21.15.173:6792]" 2021-03-09T09:34:44.725 INFO:teuthology.orchestra.run.smithi173.stdout: mon client directed command retry = 5 2021-03-09T09:34:44.725 INFO:teuthology.orchestra.run.smithi173.stdout: mon election default strategy = 1 2021-03-09T09:34:44.725 INFO:teuthology.orchestra.run.smithi173.stdout: ms bind msgr2 = False 2021-03-09T09:34:44.725 INFO:teuthology.orchestra.run.smithi173.stdout: ms inject socket failures = 5000 2021-03-09T09:34:44.726 INFO:teuthology.orchestra.run.smithi173.stdout: ms type = async 2021-03-09T09:34:44.726 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:34:44.726 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.726 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2021-03-09T09:34:44.727 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2021-03-09T09:34:44.727 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.727 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2021-03-09T09:34:44.727 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 600 2021-03-09T09:34:44.728 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.728 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2021-03-09T09:34:44.728 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2021-03-09T09:34:44.728 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.728 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2021-03-09T09:34:44.729 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2021-03-09T09:34:44.729 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2021-03-09T09:34:44.729 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.729 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2021-03-09T09:34:44.730 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2021-03-09T09:34:44.730 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.730 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2021-03-09T09:34:44.731 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.731 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2021-03-09T09:34:44.731 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.731 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2021-03-09T09:34:44.731 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.732 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2021-03-09T09:34:44.732 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2021-03-09T09:34:44.732 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.732 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2021-03-09T09:34:44.732 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2021-03-09T09:34:44.733 INFO:teuthology.orchestra.run.smithi173.stdout: bdev async discard = True 2021-03-09T09:34:44.733 INFO:teuthology.orchestra.run.smithi173.stdout: bdev enable discard = True 2021-03-09T09:34:44.733 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore allocator = bitmap 2021-03-09T09:34:44.733 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2021-03-09T09:34:44.734 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2021-03-09T09:34:44.734 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 20 2021-03-09T09:34:44.734 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 20 2021-03-09T09:34:44.734 INFO:teuthology.orchestra.run.smithi173.stdout: debug filestore = 20 2021-03-09T09:34:44.735 INFO:teuthology.orchestra.run.smithi173.stdout: debug journal = 20 2021-03-09T09:34:44.735 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 20 2021-03-09T09:34:44.735 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 25 2021-03-09T09:34:44.735 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 10 2021-03-09T09:34:44.736 INFO:teuthology.orchestra.run.smithi173.stdout: mon client ping interval = 4 2021-03-09T09:34:44.736 INFO:teuthology.orchestra.run.smithi173.stdout: mon client ping timeout = 12 2021-03-09T09:34:44.736 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2021-03-09T09:34:44.736 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2021-03-09T09:34:44.736 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2021-03-09T09:34:44.737 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify cached snaps = True 2021-03-09T09:34:44.737 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify missing on start = True 2021-03-09T09:34:44.737 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2021-03-09T09:34:44.738 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2021-03-09T09:34:44.738 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue = debug_random 2021-03-09T09:34:44.738 INFO:teuthology.orchestra.run.smithi173.stdout: osd op queue cut off = debug_random 2021-03-09T09:34:44.739 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.739 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2021-03-09T09:34:44.739 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2021-03-09T09:34:44.739 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2021-03-09T09:34:44.739 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2021-03-09T09:34:44.740 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2021-03-09T09:34:44.740 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2021-03-09T09:34:44.740 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2021-03-09T09:34:44.740 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2021-03-09T09:34:44.741 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.741 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2021-03-09T09:34:44.741 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2021-03-09T09:34:44.741 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2021-03-09T09:34:44.742 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2021-03-09T09:34:44.742 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2021-03-09T09:34:44.742 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2021-03-09T09:34:44.742 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2021-03-09T09:34:44.742 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2021-03-09T09:34:44.743 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2021-03-09T09:34:44.743 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2021-03-09T09:34:44.743 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2021-03-09T09:34:44.743 INFO:teuthology.orchestra.run.smithi173.stdout: mon min osdmap epochs = 25 2021-03-09T09:34:44.743 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune interval = 2 2021-03-09T09:34:44.744 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune min = 15 2021-03-09T09:34:44.744 INFO:teuthology.orchestra.run.smithi173.stdout: mon osdmap full prune txsize = 2 2021-03-09T09:34:44.744 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub inject crc mismatch = 0.01 2021-03-09T09:34:44.744 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub inject missing keys = 0.05 2021-03-09T09:34:44.745 INFO:teuthology.orchestra.run.smithi173.stdout: mon scrub interval = 300 2021-03-09T09:34:44.745 INFO:teuthology.orchestra.run.smithi173.stdout: paxos service trim min = 5 2021-03-09T09:34:44.745 INFO:teuthology.orchestra.run.smithi173.stdout: 2021-03-09T09:34:44.745 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2021-03-09T09:34:44.745 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2021-03-09T09:34:44.746 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2021-03-09T09:34:44.746 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2021-03-09T09:34:44.746 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-09T09:34:44.746 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-09T09:34:44.746 INFO:teuthology.orchestra.run.smithi173.stdout: debug monc = 20 2021-03-09T09:34:44.747 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2021-03-09T09:34:44.747 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2021-03-09T09:34:44.747 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.b] 2021-03-09T09:34:44.747 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.c] 2021-03-09T09:34:44.748 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.d] 2021-03-09T09:34:44.748 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.e] 2021-03-09T09:34:44.748 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.f] 2021-03-09T09:34:44.749 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.g] 2021-03-09T09:34:44.749 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.h] 2021-03-09T09:34:44.749 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.i] 2021-03-09T09:34:44.752 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2021-03-09T09:34:44.752 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2021-03-09T09:34:44.752 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-09T09:34:44.752 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2021-03-09T09:34:44.753 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.753 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2021-03-09T09:34:44.753 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.753 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2021-03-09T09:34:44.753 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.754 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2021-03-09T09:34:44.754 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2021-03-09T09:34:44.754 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.754 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2021-03-09T09:34:44.755 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2021-03-09T09:34:44.755 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.755 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-09T09:34:44.755 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2021-03-09T09:34:44.756 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.756 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2021-03-09T09:34:44.756 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-09T09:34:44.756 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.757 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2021-03-09T09:34:44.757 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.757 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2021-03-09T09:34:44.758 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2021-03-09T09:34:44.758 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2021-03-09T09:34:44.758 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2021-03-09T09:34:44.758 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2021-03-09T09:34:44.758 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2021-03-09T09:34:44.759 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2021-03-09T09:34:44.759 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-09T09:34:44.759 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-09T09:34:44.759 INFO:teuthology.orchestra.run.smithi143.stdout: mon_allow_pool_size_one = true 2021-03-09T09:34:44.760 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.760 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-09T09:34:44.760 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.760 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2021-03-09T09:34:44.760 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.761 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2021-03-09T09:34:44.761 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.761 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2021-03-09T09:34:44.761 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2021-03-09T09:34:44.761 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2021-03-09T09:34:44.762 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "[v1:172.21.15.143:6789],[v1:172.21.15.143:6790],[v1:172.21.15.143:6791],[v1:172.21.15.143:6792],[v1:172.21.15.143:6793],[v1:172.21.15.173:6789],[v1:172.21.15.173:6790],[v1:172.21.15.173:6791],[v1:172.21.15.173:6792]" 2021-03-09T09:34:44.762 INFO:teuthology.orchestra.run.smithi143.stdout: mon client directed command retry = 5 2021-03-09T09:34:44.762 INFO:teuthology.orchestra.run.smithi143.stdout: mon election default strategy = 1 2021-03-09T09:34:44.763 INFO:teuthology.orchestra.run.smithi143.stdout: ms bind msgr2 = False 2021-03-09T09:34:44.763 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject socket failures = 5000 2021-03-09T09:34:44.763 INFO:teuthology.orchestra.run.smithi143.stdout: ms type = async 2021-03-09T09:34:44.763 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:34:44.764 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.764 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2021-03-09T09:34:44.764 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2021-03-09T09:34:44.765 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.765 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2021-03-09T09:34:44.766 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2021-03-09T09:34:44.766 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.766 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2021-03-09T09:34:44.767 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2021-03-09T09:34:44.767 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.768 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2021-03-09T09:34:44.768 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2021-03-09T09:34:44.768 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2021-03-09T09:34:44.769 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.769 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2021-03-09T09:34:44.769 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2021-03-09T09:34:44.770 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.770 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2021-03-09T09:34:44.771 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.771 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2021-03-09T09:34:44.771 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.771 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2021-03-09T09:34:44.772 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.772 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2021-03-09T09:34:44.772 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2021-03-09T09:34:44.772 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.772 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2021-03-09T09:34:44.773 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2021-03-09T09:34:44.773 INFO:teuthology.orchestra.run.smithi143.stdout: bdev async discard = True 2021-03-09T09:34:44.773 INFO:teuthology.orchestra.run.smithi143.stdout: bdev enable discard = True 2021-03-09T09:34:44.773 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore allocator = bitmap 2021-03-09T09:34:44.774 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2021-03-09T09:34:44.774 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2021-03-09T09:34:44.774 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 20 2021-03-09T09:34:44.774 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 20 2021-03-09T09:34:44.775 INFO:teuthology.orchestra.run.smithi143.stdout: debug filestore = 20 2021-03-09T09:34:44.775 INFO:teuthology.orchestra.run.smithi143.stdout: debug journal = 20 2021-03-09T09:34:44.775 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 20 2021-03-09T09:34:44.776 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 25 2021-03-09T09:34:44.776 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 10 2021-03-09T09:34:44.776 INFO:teuthology.orchestra.run.smithi143.stdout: mon client ping interval = 4 2021-03-09T09:34:44.776 INFO:teuthology.orchestra.run.smithi143.stdout: mon client ping timeout = 12 2021-03-09T09:34:44.776 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2021-03-09T09:34:44.777 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2021-03-09T09:34:44.777 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2021-03-09T09:34:44.777 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify cached snaps = True 2021-03-09T09:34:44.777 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify missing on start = True 2021-03-09T09:34:44.778 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2021-03-09T09:34:44.778 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2021-03-09T09:34:44.778 INFO:teuthology.orchestra.run.smithi143.stdout: osd op queue = debug_random 2021-03-09T09:34:44.778 INFO:teuthology.orchestra.run.smithi143.stdout: osd op queue cut off = debug_random 2021-03-09T09:34:44.778 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.779 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2021-03-09T09:34:44.779 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2021-03-09T09:34:44.779 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2021-03-09T09:34:44.779 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2021-03-09T09:34:44.780 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2021-03-09T09:34:44.780 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2021-03-09T09:34:44.780 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2021-03-09T09:34:44.780 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2021-03-09T09:34:44.781 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.781 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2021-03-09T09:34:44.781 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2021-03-09T09:34:44.781 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2021-03-09T09:34:44.781 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2021-03-09T09:34:44.782 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2021-03-09T09:34:44.782 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2021-03-09T09:34:44.782 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2021-03-09T09:34:44.782 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2021-03-09T09:34:44.783 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2021-03-09T09:34:44.783 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2021-03-09T09:34:44.783 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2021-03-09T09:34:44.783 INFO:teuthology.orchestra.run.smithi143.stdout: mon min osdmap epochs = 25 2021-03-09T09:34:44.783 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune interval = 2 2021-03-09T09:34:44.784 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune min = 15 2021-03-09T09:34:44.784 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune txsize = 2 2021-03-09T09:34:44.784 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub inject crc mismatch = 0.01 2021-03-09T09:34:44.784 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub inject missing keys = 0.05 2021-03-09T09:34:44.784 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub interval = 300 2021-03-09T09:34:44.785 INFO:teuthology.orchestra.run.smithi143.stdout: paxos service trim min = 5 2021-03-09T09:34:44.785 INFO:teuthology.orchestra.run.smithi143.stdout: 2021-03-09T09:34:44.785 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2021-03-09T09:34:44.785 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2021-03-09T09:34:44.785 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2021-03-09T09:34:44.786 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2021-03-09T09:34:44.786 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-09T09:34:44.786 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-09T09:34:44.786 INFO:teuthology.orchestra.run.smithi143.stdout: debug monc = 20 2021-03-09T09:34:44.787 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2021-03-09T09:34:44.787 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2021-03-09T09:34:44.787 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2021-03-09T09:34:44.787 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2021-03-09T09:34:44.787 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.d] 2021-03-09T09:34:44.788 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.e] 2021-03-09T09:34:44.788 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.f] 2021-03-09T09:34:44.788 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.g] 2021-03-09T09:34:44.788 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.h] 2021-03-09T09:34:44.788 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.i] 2021-03-09T09:34:44.789 INFO:tasks.ceph:Creating admin key on mon.a... 2021-03-09T09:34:44.789 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-03-09T09:34:44.900 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-09T09:34:44.900 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:34:44.900 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2021-03-09T09:34:44.906 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:34:44.907 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2021-03-09T09:34:44.955 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi143.front.sepia.ceph.com 2021-03-09T09:34:44.956 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:34:44.956 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.keyring 2021-03-09T09:34:44.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T09:34:45.021 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:34:45.022 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:34:45.068 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2021-03-09T09:34:45.068 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:34:45.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.keyring 2021-03-09T09:34:45.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T09:34:45.094 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:34:45.094 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:34:45.143 INFO:tasks.ceph:Setting up mon nodes... 2021-03-09T09:34:45.144 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-09T09:34:45.144 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-03-09T09:34:45.282 INFO:teuthology.orchestra.run.smithi173.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-03-09T09:34:45.288 INFO:tasks.ceph:Setting up mds nodes... 2021-03-09T09:34:45.288 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-09T09:34:45.289 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-03-09T09:34:45.368 INFO:teuthology.orchestra.run.smithi173.stdout:creating /etc/ceph/ceph.client.0.keyring 2021-03-09T09:34:45.383 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-09T09:34:45.384 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2021-03-09T09:34:45.384 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-03-09T09:34:45.399 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-03-09T09:34:45.400 INFO:tasks.ceph:role: osd.0 2021-03-09T09:34:45.400 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com 2021-03-09T09:34:45.400 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-09T09:34:45.935 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:34:45.936 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:34:45.936 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:34:45.936 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:34:45.936 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:34:45.937 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:34:45.937 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:34:45.937 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:34:45.937 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:34:45.939 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2021-03-09T09:34:45.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-03-09T09:34:46.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-03-09T09:34:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:34:46.024 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:34:46.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-03-09T09:34:46.079 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-03-09T09:34:46.080 INFO:tasks.ceph:role: osd.1 2021-03-09T09:34:46.080 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com 2021-03-09T09:34:46.081 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-09T09:34:46.613 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:34:46.613 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:34:46.614 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:34:46.614 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:34:46.614 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:34:46.614 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:34:46.615 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:34:46.615 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:34:46.615 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:34:46.616 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2021-03-09T09:34:46.617 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-03-09T09:34:46.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-03-09T09:34:46.676 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:34:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:34:46.678 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-03-09T09:34:46.740 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-03-09T09:34:46.741 INFO:tasks.ceph:role: osd.2 2021-03-09T09:34:46.741 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com 2021-03-09T09:34:46.742 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-09T09:34:47.237 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:34:47.238 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:34:47.238 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:34:47.238 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:34:47.238 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:34:47.239 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:34:47.239 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:34:47.239 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:34:47.239 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:34:47.241 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2021-03-09T09:34:47.241 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-03-09T09:34:47.289 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-03-09T09:34:47.299 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:34:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:34:47.300 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:34:47.479 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:47.478+0000 7f3a2125af00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-03-09T09:34:47.480 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:47.478+0000 7f3a2125af00 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-03-09T09:34:47.480 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:47.478+0000 7f3a2125af00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T09:34:47.481 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:47.478+0000 7f3a2125af00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T09:34:47.481 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:47.478+0000 7f3a2125af00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T09:34:47.481 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:47.478+0000 7f3a2125af00 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-03-09T09:34:50.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-03-09T09:34:50.467 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:34:50.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:50.606+0000 7ff07cbb2f00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-03-09T09:34:50.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:50.606+0000 7ff07cbb2f00 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-03-09T09:34:50.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:50.606+0000 7ff07cbb2f00 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T09:34:50.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:50.606+0000 7ff07cbb2f00 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T09:34:50.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:50.606+0000 7ff07cbb2f00 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T09:34:50.610 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:50.606+0000 7ff07cbb2f00 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-03-09T09:34:53.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-03-09T09:34:53.588 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:34:53.727 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:53.726+0000 7f8dc7397f00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-03-09T09:34:53.727 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:53.726+0000 7f8dc7397f00 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-03-09T09:34:53.728 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:53.726+0000 7f8dc7397f00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T09:34:53.728 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:53.726+0000 7f8dc7397f00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T09:34:53.728 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:53.726+0000 7f8dc7397f00 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T09:34:53.729 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:34:53.726+0000 7f8dc7397f00 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-03-09T09:34:56.708 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-03-09T09:34:56.723 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-03-09T09:34:56.739 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2021-03-09T09:34:56.739 INFO:tasks.ceph:role: osd.3 2021-03-09T09:34:56.739 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com 2021-03-09T09:34:56.740 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-09T09:34:57.266 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:34:57.267 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:34:57.267 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:34:57.267 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:34:57.268 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:34:57.268 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:34:57.268 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:34:57.269 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:34:57.269 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:34:57.270 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2021-03-09T09:34:57.270 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2021-03-09T09:34:57.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-03-09T09:34:57.372 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:34:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:34:57.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2021-03-09T09:34:57.432 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2021-03-09T09:34:57.432 INFO:tasks.ceph:role: osd.4 2021-03-09T09:34:57.433 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com 2021-03-09T09:34:57.433 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-09T09:34:57.958 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:34:57.958 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:34:57.958 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:34:57.959 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:34:57.959 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:34:57.959 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:34:57.959 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:34:57.960 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:34:57.960 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:34:57.961 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2021-03-09T09:34:57.961 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2021-03-09T09:34:58.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2021-03-09T09:34:58.016 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:34:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:34:58.017 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2021-03-09T09:34:58.076 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2021-03-09T09:34:58.076 INFO:tasks.ceph:role: osd.5 2021-03-09T09:34:58.077 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com 2021-03-09T09:34:58.077 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-09T09:34:58.578 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T09:34:58.579 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T09:34:58.579 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-09T09:34:58.579 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T09:34:58.579 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2021-03-09T09:34:58.579 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-09T09:34:58.580 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T09:34:58.580 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T09:34:58.580 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T09:34:58.581 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2021-03-09T09:34:58.582 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2021-03-09T09:34:58.639 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2021-03-09T09:34:58.651 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: /sbin/restorecon: command not found 2021-03-09T09:34:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T09:34:58.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:34:58.801 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:34:58.801+0000 7f67a2bacf00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-03-09T09:34:58.802 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:34:58.801+0000 7f67a2bacf00 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-03-09T09:34:58.802 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:34:58.801+0000 7f67a2bacf00 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T09:34:58.803 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:34:58.801+0000 7f67a2bacf00 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T09:34:58.803 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:34:58.801+0000 7f67a2bacf00 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T09:34:58.804 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:34:58.801+0000 7f67a2bacf00 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-03-09T09:35:01.788 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-03-09T09:35:01.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:35:01.941 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:01.937+0000 7fde9d499f00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2021-03-09T09:35:01.942 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:01.937+0000 7fde9d499f00 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2021-03-09T09:35:01.943 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:01.937+0000 7fde9d499f00 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-03-09T09:35:01.943 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:01.937+0000 7fde9d499f00 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-03-09T09:35:01.943 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:01.937+0000 7fde9d499f00 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-03-09T09:35:01.944 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:01.941+0000 7fde9d499f00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2021-03-09T09:35:04.924 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-03-09T09:35:04.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:35:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:05.081+0000 7f4556e42f00 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2021-03-09T09:35:05.083 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:05.081+0000 7f4556e42f00 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2021-03-09T09:35:05.084 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:05.081+0000 7f4556e42f00 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-03-09T09:35:05.084 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:05.081+0000 7f4556e42f00 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-03-09T09:35:05.084 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:05.081+0000 7f4556e42f00 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-03-09T09:35:05.085 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:05.081+0000 7f4556e42f00 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2021-03-09T09:35:08.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-03-09T09:35:08.086 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-09T09:35:08.087 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:35:08.087 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2021-03-09T09:35:08.103 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:35:08.103 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2021-03-09T09:35:08.159 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:35:08.160 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2021-03-09T09:35:08.219 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:35:08.220 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2021-03-09T09:35:08.235 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:35:08.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2021-03-09T09:35:08.291 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:35:08.292 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2021-03-09T09:35:08.345 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:35:08.346 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2021-03-09T09:35:08.403 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:35:08.403 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2021-03-09T09:35:08.451 INFO:tasks.ceph:Adding keys to all mons... 2021-03-09T09:35:08.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo tee -a /etc/ceph/ceph.keyring 2021-03-09T09:35:08.456 DEBUG:teuthology.orchestra.run.smithi173:> sudo tee -a /etc/ceph/ceph.keyring 2021-03-09T09:35:08.496 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.0] 2021-03-09T09:35:08.496 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA3QUdg01S2HBAARPoegOHHF7Yg9XTRxt+P0A== 2021-03-09T09:35:08.497 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.1] 2021-03-09T09:35:08.497 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA6QUdgecZjJBAA4tbNKiUII0MkTJjFaZO2pA== 2021-03-09T09:35:08.497 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.2] 2021-03-09T09:35:08.497 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA9QUdgRBFxKxAAinNk7OgWnIhIXCkWKdBFPg== 2021-03-09T09:35:08.498 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.x] 2021-03-09T09:35:08.498 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA1QUdgPrD4EBAAx3Jr5CZECUS05+aEl0bgTQ== 2021-03-09T09:35:08.498 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.3] 2021-03-09T09:35:08.498 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBCQUdgttTvLxAAioS/Yi5M74QYErylOeKgOA== 2021-03-09T09:35:08.498 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.4] 2021-03-09T09:35:08.499 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBFQUdgyok9OBAAde3ktxuRrIWPR2xu+iPHuA== 2021-03-09T09:35:08.499 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.5] 2021-03-09T09:35:08.499 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQBJQUdgsQ0aBRAAro2PPqWCFUbHU/f62V3Q3Q== 2021-03-09T09:35:08.499 INFO:teuthology.orchestra.run.smithi143.stdout:[client.0] 2021-03-09T09:35:08.500 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQA1QUdg3T8cFhAAsdv9oABFVgN9prUF6EyGKA== 2021-03-09T09:35:08.506 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.0] 2021-03-09T09:35:08.506 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA3QUdg01S2HBAARPoegOHHF7Yg9XTRxt+P0A== 2021-03-09T09:35:08.507 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.1] 2021-03-09T09:35:08.507 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA6QUdgecZjJBAA4tbNKiUII0MkTJjFaZO2pA== 2021-03-09T09:35:08.507 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.2] 2021-03-09T09:35:08.507 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA9QUdgRBFxKxAAinNk7OgWnIhIXCkWKdBFPg== 2021-03-09T09:35:08.507 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.x] 2021-03-09T09:35:08.508 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA1QUdgPrD4EBAAx3Jr5CZECUS05+aEl0bgTQ== 2021-03-09T09:35:08.508 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.3] 2021-03-09T09:35:08.508 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBCQUdgttTvLxAAioS/Yi5M74QYErylOeKgOA== 2021-03-09T09:35:08.508 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.4] 2021-03-09T09:35:08.509 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBFQUdgyok9OBAAde3ktxuRrIWPR2xu+iPHuA== 2021-03-09T09:35:08.509 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.5] 2021-03-09T09:35:08.509 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBJQUdgsQ0aBRAAro2PPqWCFUbHU/f62V3Q3Q== 2021-03-09T09:35:08.509 INFO:teuthology.orchestra.run.smithi173.stdout:[client.0] 2021-03-09T09:35:08.510 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA1QUdg3T8cFhAAsdv9oABFVgN9prUF6EyGKA== 2021-03-09T09:35:08.511 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:08.547 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:08.655 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:08.658 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:08.755 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:08.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:08.853 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T09:35:08.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T09:35:08.951 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:08.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:09.048 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:09.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:09.149 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:09.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T09:35:09.244 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T09:35:09.247 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T09:35:09.383 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-09T09:35:09.384 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-03-09T09:35:09.399 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:35:09.557 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-03-09T09:35:09.571 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-03-09T09:35:09.627 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:35:09.790 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-03-09T09:35:09.805 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-03-09T09:35:09.858 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:35:10.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-03-09T09:35:10.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2021-03-09T09:35:10.092 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:35:10.256 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2021-03-09T09:35:10.270 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2021-03-09T09:35:10.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:35:10.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2021-03-09T09:35:10.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2021-03-09T09:35:10.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:35:10.689 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2021-03-09T09:35:10.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2021-03-09T09:35:10.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:35:10.945 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2021-03-09T09:35:10.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2021-03-09T09:35:11.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:35:11.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2021-03-09T09:35:11.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2021-03-09T09:35:11.254 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T09:35:11.421 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2021-03-09T09:35:11.436 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:35:11.439 DEBUG:teuthology.orchestra.run.smithi173:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-03-09T09:35:11.482 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-03-09T09:35:11.482 INFO:tasks.ceph.mon.a:Restarting daemon 2021-03-09T09:35:11.484 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-03-09T09:35:11.488 INFO:tasks.ceph.mon.a:Started 2021-03-09T09:35:11.489 INFO:tasks.ceph.mon.b:Restarting daemon 2021-03-09T09:35:11.489 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-03-09T09:35:11.491 INFO:tasks.ceph.mon.b:Started 2021-03-09T09:35:11.492 INFO:tasks.ceph.mon.c:Restarting daemon 2021-03-09T09:35:11.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-03-09T09:35:11.495 INFO:tasks.ceph.mon.c:Started 2021-03-09T09:35:11.495 INFO:tasks.ceph.mon.d:Restarting daemon 2021-03-09T09:35:11.496 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2021-03-09T09:35:11.498 INFO:tasks.ceph.mon.d:Started 2021-03-09T09:35:11.498 INFO:tasks.ceph.mon.e:Restarting daemon 2021-03-09T09:35:11.499 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2021-03-09T09:35:11.501 INFO:tasks.ceph.mon.e:Started 2021-03-09T09:35:11.501 INFO:tasks.ceph.mon.f:Restarting daemon 2021-03-09T09:35:11.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2021-03-09T09:35:11.526 INFO:tasks.ceph.mon.f:Started 2021-03-09T09:35:11.527 INFO:tasks.ceph.mon.g:Restarting daemon 2021-03-09T09:35:11.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2021-03-09T09:35:11.529 INFO:tasks.ceph.mon.g:Started 2021-03-09T09:35:11.530 INFO:tasks.ceph.mon.h:Restarting daemon 2021-03-09T09:35:11.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2021-03-09T09:35:11.532 INFO:tasks.ceph.mon.h:Started 2021-03-09T09:35:11.533 INFO:tasks.ceph.mon.i:Restarting daemon 2021-03-09T09:35:11.533 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2021-03-09T09:35:11.536 INFO:tasks.ceph.mon.i:Started 2021-03-09T09:35:11.536 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-03-09T09:35:11.536 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-03-09T09:35:11.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-03-09T09:35:11.539 INFO:tasks.ceph.mgr.x:Started 2021-03-09T09:35:11.540 INFO:tasks.ceph:Setting crush tunables to default 2021-03-09T09:35:11.540 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd crush tunables default 2021-03-09T09:35:11.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient: get_monmap_and_config 2021-03-09T09:35:11.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient: build_initial_monmap 2021-03-09T09:35:11.824 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient: monmap: 2021-03-09T09:35:11.824 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2021-03-09T09:35:11.824 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:11.824 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:35:11.825896+0000 2021-03-09T09:35:11.825 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:35:11.825896+0000 2021-03-09T09:35:11.825 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:11.825 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:11.826 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:11.826 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:11.826 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:11.826 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:11.827 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:11.827 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:11.827 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:11.827 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:11.827 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:11.828 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:11.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 1 Processor -- start 2021-03-09T09:35:11.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 1 -- start start 2021-03-09T09:35:11.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient: init 2021-03-09T09:35:11.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:11.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient: _add_conns ranks=[4,6,8,1,2,0,7,5,3] 2021-03-09T09:35:11.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient(hunting): picked mon.noname-c con 0x7f6c6c0981b0 addr v1:172.21.15.143:6791/0 2021-03-09T09:35:11.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient(hunting): picked mon.noname-d con 0x7f6c6c099bf0 addr v1:172.21.15.143:6792/0 2021-03-09T09:35:11.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient(hunting): picked mon.noname-e con 0x7f6c6c09a070 addr v1:172.21.15.143:6793/0 2021-03-09T09:35:11.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 1 -- --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c0460b0 con 0x7f6c6c0981b0 2021-03-09T09:35:11.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 1 -- --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c09b020 con 0x7f6c6c099bf0 2021-03-09T09:35:11.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 1 -- --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c090490 con 0x7f6c6c09a070 2021-03-09T09:35:11.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:11.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c72467700 10 monclient(hunting): authenticate will time out at 701.231530s 2021-03-09T09:35:11.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c70a04700 1 --1- >> v1:172.21.15.143:6793/0 conn(0x7f6c6c09a070 0x7f6c6c134120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6793/0 says I am v1:172.21.15.143:44040/0 (socket says 172.21.15.143:44040) 2021-03-09T09:35:11.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c6bfff700 1 --1- >> v1:172.21.15.143:6791/0 conn(0x7f6c6c0981b0 0x7f6c6c0994e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6791/0 says I am v1:172.21.15.143:37340/0 (socket says 172.21.15.143:37340) 2021-03-09T09:35:11.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:11.822+0000 7f6c70a04700 1 -- 172.21.15.143:0/4141712820 learned_addr learned my addr 172.21.15.143:0/4141712820 (peer_addr_for_me v1:172.21.15.143:0/0) 2021-03-09T09:35:14.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 10 monclient(hunting): tick 2021-03-09T09:35:14.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 1 monclient(hunting): continuing hunt 2021-03-09T09:35:14.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:35:14.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6793/0 conn(0x7f6c6c09a070 legacy=0x7f6c6c134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:14.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6792/0 conn(0x7f6c6c099bf0 legacy=0x7f6c6c1309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:14.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6791/0 conn(0x7f6c6c0981b0 legacy=0x7f6c6c0994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:14.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 10 monclient: _add_conns ranks=[1,5,0,2,7,3] 2021-03-09T09:35:14.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 10 monclient(hunting): picked mon.noname-f con 0x7f6c4c0034c0 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:14.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 10 monclient(hunting): picked mon.noname-h con 0x7f6c4c007130 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:14.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f6c4c00aec0 addr v1:172.21.15.143:6789/0 2021-03-09T09:35:14.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c090730 con 0x7f6c4c00aec0 2021-03-09T09:35:14.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c00fcd0 con 0x7f6c4c0034c0 2021-03-09T09:35:14.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c010f20 con 0x7f6c4c007130 2021-03-09T09:35:14.829 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:14.826+0000 7f6c69ffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:17.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 10 monclient(hunting): tick 2021-03-09T09:35:17.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 1 monclient(hunting): continuing hunt 2021-03-09T09:35:17.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:35:17.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.173:6791/0 conn(0x7f6c4c007130 legacy=0x7f6c4c0095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:17.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.173:6789/0 conn(0x7f6c4c0034c0 legacy=0x7f6c4c005910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:17.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6789/0 conn(0x7f6c4c00aec0 legacy=0x7f6c4c00d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:17.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c70a04700 1 -- 172.21.15.143:0/4141712820 reap_dead start 2021-03-09T09:35:17.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 10 monclient: _add_conns ranks=[7,2,3] 2021-03-09T09:35:17.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c70a04700 1 -- 172.21.15.143:0/4141712820 reap_dead start 2021-03-09T09:35:17.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 10 monclient(hunting): picked mon.noname-i con 0x7f6c4c011300 addr v1:172.21.15.173:6792/0 2021-03-09T09:35:17.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f6c4c0117e0 addr v1:172.21.15.143:6790/0 2021-03-09T09:35:17.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 10 monclient(hunting): picked mon.noname-g con 0x7f6c4c00fe60 addr v1:172.21.15.173:6790/0 2021-03-09T09:35:17.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c012230 con 0x7f6c4c0117e0 2021-03-09T09:35:17.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c010a20 con 0x7f6c4c00fe60 2021-03-09T09:35:17.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c010bb0 con 0x7f6c4c011300 2021-03-09T09:35:17.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c69ffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:17.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c6b7fe700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6790/0 conn(0x7f6c4c0117e0 legacy=0x7f6c4c005910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T09:35:17.836 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c6b7fe700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6790/0 conn(0x7f6c4c0117e0 legacy=0x7f6c4c005910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T09:35:17.836 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:17.826+0000 7f6c6b7fe700 1 --1- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6790/0 conn(0x7f6c4c0117e0 0x7f6c4c005910 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2021-03-09T09:35:20.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 10 monclient(hunting): tick 2021-03-09T09:35:20.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 1 monclient(hunting): continuing hunt 2021-03-09T09:35:20.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:35:20.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.173:6792/0 conn(0x7f6c4c011300 legacy=0x7f6c4c0095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:20.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.173:6790/0 conn(0x7f6c4c00fe60 legacy=0x7f6c4c010310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:20.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6790/0 conn(0x7f6c4c0117e0 legacy=0x7f6c4c005910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:20.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 10 monclient: _add_conns ranks=[4,6,1,0,8,3,7,2,5] 2021-03-09T09:35:20.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f6c4c00d5f0 addr v1:172.21.15.143:6791/0 2021-03-09T09:35:20.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 10 monclient(hunting): picked mon.noname-d con 0x7f6c4c00e2c0 addr v1:172.21.15.143:6792/0 2021-03-09T09:35:20.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 10 monclient(hunting): picked mon.noname-f con 0x7f6c4c018c70 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:20.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c00d2f0 con 0x7f6c4c018c70 2021-03-09T09:35:20.829 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c00e980 con 0x7f6c4c00d5f0 2021-03-09T09:35:20.829 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4c01e830 con 0x7f6c4c00e2c0 2021-03-09T09:35:20.829 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:20.826+0000 7f6c69ffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:21.663 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.662+0000 7f6c6bfff700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6791/0 conn(0x7f6c4c00d5f0 legacy=0x7f6c4c00db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2021-03-09T09:35:21.664 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.662+0000 7f6c6bfff700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6791/0 conn(0x7f6c4c00d5f0 legacy=0x7f6c4c00db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:21.664 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.662+0000 7f6c6bfff700 1 --1- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6791/0 conn(0x7f6c4c00d5f0 0x7f6c4c00db10 :-1 s=OPENED pgs=33 cs=1 l=1).handle_message read tag failed 2021-03-09T09:35:21.664 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.662+0000 7f6c6bfff700 1 --1- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6791/0 conn(0x7f6c4c00d5f0 0x7f6c4c00db10 :-1 s=OPENED pgs=33 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:35:21.664 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.662+0000 7f6c6affd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6791/0 2021-03-09T09:35:21.936 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867904883 0 0) 0x7f6c4c00d2f0 con 0x7f6c4c018c70 2021-03-09T09:35:21.937 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:21.937 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 10 monclient(hunting): my global_id is 4114 2021-03-09T09:35:21.937 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c58003230 con 0x7f6c4c018c70 2021-03-09T09:35:21.937 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 <== mon.6 v1:172.21.15.143:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397618430 0 0) 0x7f6c4c01e830 con 0x7f6c4c00e2c0 2021-03-09T09:35:21.938 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:21.938 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 10 monclient(hunting): my global_id is 4119 2021-03-09T09:35:21.938 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.143:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c4c00d2f0 con 0x7f6c4c00e2c0 2021-03-09T09:35:21.940 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4244607902 0 0) 0x7f6c58003230 con 0x7f6c4c018c70 2021-03-09T09:35:21.940 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6792/0 conn(0x7f6c4c00e2c0 legacy=0x7f6c4c017550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:21.940 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.143:6791/0 conn(0x7f6c4c00d5f0 legacy=0x7f6c4c00db10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:21.940 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 10 monclient: _finish_hunting 0 2021-03-09T09:35:21.941 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 1 monclient: found mon.noname-f 2021-03-09T09:35:21.941 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.934+0000 7f6c6affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:21.941 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.173:6789/0 2021-03-09T09:35:21.941 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c01fa60 con 0x7f6c4c018c70 2021-03-09T09:35:21.941 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 10 monclient: _finish_auth 0 2021-03-09T09:35:21.942 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:21.942 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f6c5c001720 con 0x7f6c4c018c70 2021-03-09T09:35:21.942 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:21.942 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2021-03-09T09:35:21.942 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 10 monclient: dump: 2021-03-09T09:35:21.943 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2021-03-09T09:35:21.943 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:21.943 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:21.943 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:21.943 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:21.944 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:21.944 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:21.944 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:21.944 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:21.945 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:21.945 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:21.945 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:21.945 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:21.945 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:21.946 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:21.946 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:21.946 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 5 monclient: authenticate success, global_id 4114 2021-03-09T09:35:21.946 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:21.946 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c70a04700 1 -- 172.21.15.143:0/4141712820 reap_dead start 2021-03-09T09:35:21.947 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c5c001970 con 0x7f6c4c018c70 2021-03-09T09:35:21.947 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:21.947 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 1 -- 172.21.15.143:0/4141712820 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f6c5c0045b0 con 0x7f6c4c018c70 2021-03-09T09:35:21.947 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:21.947 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 10 monclient: got monmap 1 from mon.f (according to old e1) 2021-03-09T09:35:21.948 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c6affd700 10 monclient: dump: 2021-03-09T09:35:21.948 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2021-03-09T09:35:21.948 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:21.948 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:21.948 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:21.949 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:21.949 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:21.949 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:21.949 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:21.949 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:21.950 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:21.950 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:21.950 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:21.950 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:21.950 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:21.951 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:21.951 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:21.951 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:21.951 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 4 set_mon_vals no callback set 2021-03-09T09:35:21.951 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 10 monclient: shutdown 2021-03-09T09:35:21.952 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:21.952 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 1 -- 172.21.15.143:0/4141712820 >> v1:172.21.15.173:6789/0 conn(0x7f6c4c018c70 legacy=0x7f6c4c01b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:21.952 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 1 -- 172.21.15.143:0/4141712820 shutdown_connections 2021-03-09T09:35:21.952 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 1 -- 172.21.15.143:0/4141712820 >> 172.21.15.143:0/4141712820 conn(0x7f6c6c09b780 msgr2=0x7f6c6c09bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:21.953 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 1 -- 172.21.15.143:0/4141712820 shutdown_connections 2021-03-09T09:35:21.953 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 1 -- 172.21.15.143:0/4141712820 wait complete. 2021-03-09T09:35:21.953 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 10 monclient: build_initial_monmap 2021-03-09T09:35:21.953 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 10 monclient: monmap: 2021-03-09T09:35:21.953 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2021-03-09T09:35:21.953 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:21.954 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2021-03-09T09:35:21.954 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2021-03-09T09:35:21.954 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:21.954 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:21.955 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:21.955 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:21.955 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:21.955 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:21.955 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:21.956 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:21.956 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:21.956 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:21.956 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:21.956 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:21.957 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 1 Processor -- start 2021-03-09T09:35:21.960 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 1 -- start start 2021-03-09T09:35:21.961 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.938+0000 7f6c72467700 10 monclient: init 2021-03-09T09:35:21.961 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:21.961 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 10 monclient: _add_conns ranks=[1,7,6,5,3,4,8,2,0] 2021-03-09T09:35:21.961 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 10 monclient(hunting): picked mon.noname-f con 0x7f6c4c018c70 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:21.962 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 10 monclient(hunting): picked mon.noname-i con 0x7f6c6c13b4f0 addr v1:172.21.15.173:6792/0 2021-03-09T09:35:21.962 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 10 monclient(hunting): picked mon.noname-d con 0x7f6c6c096040 addr v1:172.21.15.143:6792/0 2021-03-09T09:35:21.962 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c11f4b0 con 0x7f6c4c018c70 2021-03-09T09:35:21.962 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 1 -- --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c128850 con 0x7f6c6c096040 2021-03-09T09:35:21.962 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 1 -- --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c11f6e0 con 0x7f6c6c13b4f0 2021-03-09T09:35:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c72467700 10 monclient(hunting): authenticate will time out at 711.348315s 2021-03-09T09:35:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c6bfff700 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f6c4c018c70 0x7f6c6c128140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.143:54886/0 (socket says 172.21.15.143:54886) 2021-03-09T09:35:21.963 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c70a04700 1 --1- >> v1:172.21.15.143:6792/0 conn(0x7f6c6c096040 0x7f6c6c1474b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6792/0 says I am v1:172.21.15.143:52478/0 (socket says 172.21.15.143:52478) 2021-03-09T09:35:21.964 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c6bfff700 1 -- 172.21.15.143:0/3759292706 learned_addr learned my addr 172.21.15.143:0/3759292706 (peer_addr_for_me v1:172.21.15.143:0/0) 2021-03-09T09:35:21.964 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.6 v1:172.21.15.143:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180974436 0 0) 0x7f6c6c128850 con 0x7f6c6c096040 2021-03-09T09:35:21.964 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c53fff700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:21.964 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c53fff700 10 monclient(hunting): my global_id is 4128 2021-03-09T09:35:21.964 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.942+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 --> v1:172.21.15.143:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c4c00d840 con 0x7f6c6c096040 2021-03-09T09:35:21.965 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634242367 0 0) 0x7f6c6c11f4b0 con 0x7f6c4c018c70 2021-03-09T09:35:21.965 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:21.965 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient(hunting): my global_id is 4123 2021-03-09T09:35:21.965 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c6c128850 con 0x7f6c4c018c70 2021-03-09T09:35:21.965 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 482222705 0 0) 0x7f6c6c128850 con 0x7f6c4c018c70 2021-03-09T09:35:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c6c11f4b0 con 0x7f6c4c018c70 2021-03-09T09:35:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f6c60001d00 con 0x7f6c4c018c70 2021-03-09T09:35:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2021-03-09T09:35:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient(hunting): dump: 2021-03-09T09:35:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2021-03-09T09:35:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:21.968 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:21.969 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:21.970 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:21.970 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:21.970 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:21.970 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 200672418 0 0) 0x7f6c6c11f4b0 con 0x7f6c4c018c70 2021-03-09T09:35:21.970 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 >> v1:172.21.15.173:6792/0 conn(0x7f6c6c13b4f0 legacy=0x7f6c6c143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:21.971 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 >> v1:172.21.15.143:6792/0 conn(0x7f6c6c096040 legacy=0x7f6c6c1474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:21.971 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient: _finish_hunting 0 2021-03-09T09:35:21.971 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 monclient: found mon.f 2021-03-09T09:35:21.971 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:21.971 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:21.972 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c11f980 con 0x7f6c4c018c70 2021-03-09T09:35:21.972 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient: _finish_auth 0 2021-03-09T09:35:21.972 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:21.972 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c72467700 5 monclient: authenticate success, global_id 4123 2021-03-09T09:35:21.973 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c72467700 10 monclient: _renew_subs 2021-03-09T09:35:21.973 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c72467700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:21.973 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c72467700 1 -- 172.21.15.143:0/3759292706 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c11fc20 con 0x7f6c4c018c70 2021-03-09T09:35:21.973 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c72467700 10 monclient: _renew_subs 2021-03-09T09:35:21.973 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c72467700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:21.974 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c72467700 1 -- 172.21.15.143:0/3759292706 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c11da40 con 0x7f6c4c018c70 2021-03-09T09:35:21.974 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c60002000 con 0x7f6c4c018c70 2021-03-09T09:35:21.974 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:21.974 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f6c60004e60 con 0x7f6c4c018c70 2021-03-09T09:35:21.974 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:21.975 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c6a7fc700 4 set_mon_vals no callback set 2021-03-09T09:35:21.975 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient: got monmap 1 from mon.f (according to old e1) 2021-03-09T09:35:21.975 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 10 monclient: dump: 2021-03-09T09:35:21.975 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2021-03-09T09:35:21.975 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:21.976 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:21.976 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:21.976 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:21.976 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:21.976 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:21.977 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:21.977 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:21.977 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:21.977 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:21.977 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:21.978 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:21.978 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:21.978 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:21.978 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:21.979 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f6c600052a0 con 0x7f6c4c018c70 2021-03-09T09:35:21.979 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.946+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 3218538002 0 0) 0x7f6c600064c0 con 0x7f6c4c018c70 2021-03-09T09:35:21.979 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.950+0000 7f6c72467700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:21.979 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.950+0000 7f6c72467700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:21.979 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.950+0000 7f6c72467700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:21.980 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.950+0000 7f6c72467700 1 -- 172.21.15.143:0/3759292706 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c48004d60 con 0x7f6c4c018c70 2021-03-09T09:35:21.980 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.950+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (unknown 1092875540 0 637523725) 0x7f6c4c00e980 con 0x7f6c4c018c70 2021-03-09T09:35:21.980 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.950+0000 7f6c53fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:21.980 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:21.950+0000 7f6c53fff700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:22.035 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.034+0000 7f6c72467700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2021-03-09T09:35:22.035 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.034+0000 7f6c72467700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2021-03-09T09:35:22.035 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.034+0000 7f6c72467700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:22.036 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.034+0000 7f6c72467700 1 -- 172.21.15.143:0/3759292706 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6c48005010 con 0x7f6c4c018c70 2021-03-09T09:35:22.041 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c53fff700 1 -- 172.21.15.143:0/3759292706 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f6c60005840 con 0x7f6c4c018c70 2021-03-09T09:35:22.042 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c53fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2021-03-09T09:35:22.042 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c53fff700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2021-03-09T09:35:22.043 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c72467700 10 monclient: shutdown 2021-03-09T09:35:22.043 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c72467700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:22.043 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c72467700 1 -- 172.21.15.143:0/3759292706 >> v1:172.21.15.173:6789/0 conn(0x7f6c4c018c70 legacy=0x7f6c6c128140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.044 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c72467700 1 -- 172.21.15.143:0/3759292706 shutdown_connections 2021-03-09T09:35:22.044 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c72467700 1 -- 172.21.15.143:0/3759292706 >> 172.21.15.143:0/3759292706 conn(0x7f6c6c09b780 msgr2=0x7f6c6c0670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:22.044 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c72467700 1 -- 172.21.15.143:0/3759292706 shutdown_connections 2021-03-09T09:35:22.044 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:22.042+0000 7f6c72467700 1 -- 172.21.15.143:0/3759292706 wait complete. 2021-03-09T09:35:22.045 INFO:teuthology.orchestra.run.smithi143.stderr:adjusted tunables profile to default 2021-03-09T09:35:22.058 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-03-09T09:35:22.059 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:35:22.059 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2021-03-09T09:35:22.068 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:35:22.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2021-03-09T09:35:22.126 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2021-03-09T09:35:22.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2021-03-09T09:35:22.181 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:35:22.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2021-03-09T09:35:22.194 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:35:22.195 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2021-03-09T09:35:22.250 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2021-03-09T09:35:22.250 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2021-03-09T09:35:22.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 72ccc8f8-8bb0-4230-bfea-c47a8dadaf1c 0 2021-03-09T09:35:22.592 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.589+0000 7f8ef9891700 10 monclient: get_monmap_and_config 2021-03-09T09:35:22.593 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.589+0000 7f8ef9891700 10 monclient: build_initial_monmap 2021-03-09T09:35:22.593 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.589+0000 7f8ef9891700 10 monclient: monmap: 2021-03-09T09:35:22.593 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:22.593 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.593 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:35:22.595480+0000 2021-03-09T09:35:22.594 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:35:22.595480+0000 2021-03-09T09:35:22.594 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:22.594 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.594 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:22.595 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:22.595 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:22.595 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:22.595 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:22.595 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:22.596 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:22.596 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:22.596 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:22.596 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.605 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 Processor -- start 2021-03-09T09:35:22.605 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 -- start start 2021-03-09T09:35:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient: init 2021-03-09T09:35:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient: _add_conns ranks=[1,4,2,5,0,7,8,3,6] 2021-03-09T09:35:22.606 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient(hunting): picked mon.noname-f con 0x7f8ef4097f80 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient(hunting): picked mon.noname-c con 0x7f8ef4099b10 addr v1:172.21.15.143:6791/0 2021-03-09T09:35:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient(hunting): picked mon.noname-b con 0x7f8ef4099f70 addr v1:172.21.15.143:6790/0 2021-03-09T09:35:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef40460b0 con 0x7f8ef4097f80 2021-03-09T09:35:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef4090060 con 0x7f8ef4099f70 2021-03-09T09:35:22.607 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 -- --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef4090300 con 0x7f8ef4099b10 2021-03-09T09:35:22.608 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:22.608 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient(hunting): authenticate will time out at 723.024981s 2021-03-09T09:35:22.608 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef888f700 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f8ef4097f80 0x7f8ef4099400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:60480/0 (socket says 172.21.15.173:60480) 2021-03-09T09:35:22.608 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef888f700 1 -- 172.21.15.173:0/448475782 learned_addr learned my addr 172.21.15.173:0/448475782 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:22.608 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 -- 172.21.15.173:0/448475782 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935900343 0 0) 0x7f8ef40460b0 con 0x7f8ef4097f80 2021-03-09T09:35:22.609 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:22.609 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient(hunting): my global_id is 4132 2021-03-09T09:35:22.609 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 -- 172.21.15.173:0/448475782 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0003230 con 0x7f8ef4097f80 2021-03-09T09:35:22.609 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 -- 172.21.15.173:0/448475782 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 76547969 0 0) 0x7f8ee0003230 con 0x7f8ef4097f80 2021-03-09T09:35:22.609 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 -- 172.21.15.173:0/448475782 >> v1:172.21.15.143:6791/0 conn(0x7f8ef4099b10 legacy=0x7f8ef4130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.610 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 -- 172.21.15.173:0/448475782 >> v1:172.21.15.143:6790/0 conn(0x7f8ef4099f70 legacy=0x7f8ef4133ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.610 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: _finish_hunting 0 2021-03-09T09:35:22.610 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 monclient: found mon.noname-f 2021-03-09T09:35:22.610 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:22.611 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.173:6789/0 2021-03-09T09:35:22.611 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 -- 172.21.15.173:0/448475782 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef40905a0 con 0x7f8ef4097f80 2021-03-09T09:35:22.611 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: _finish_auth 0 2021-03-09T09:35:22.611 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:22.611 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 5 monclient: authenticate success, global_id 4132 2021-03-09T09:35:22.612 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 -- 172.21.15.173:0/448475782 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f8ee4001820 con 0x7f8ef4097f80 2021-03-09T09:35:22.612 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:22.612 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.612 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2021-03-09T09:35:22.613 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: dump: 2021-03-09T09:35:22.613 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.613 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.613 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.613 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.614 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.614 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.614 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.614 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.614 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.615 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.615 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.615 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.615 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.615 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.616 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.616 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.616 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:22.616 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 -- 172.21.15.173:0/448475782 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ee4001ad0 con 0x7f8ef4097f80 2021-03-09T09:35:22.616 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:22.617 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 1 -- 172.21.15.173:0/448475782 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f8ee40048f0 con 0x7f8ef4097f80 2021-03-09T09:35:22.617 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:22.617 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 4 set_mon_vals no callback set 2021-03-09T09:35:22.617 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.617 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: got monmap 1 from mon.f (according to old e1) 2021-03-09T09:35:22.618 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8eeb7fe700 10 monclient: dump: 2021-03-09T09:35:22.618 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.618 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.618 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.619 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.619 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.619 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.619 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.619 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.620 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.620 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.620 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.620 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.620 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.621 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.621 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.621 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.621 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient: shutdown 2021-03-09T09:35:22.621 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:22.622 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 -- 172.21.15.173:0/448475782 >> v1:172.21.15.173:6789/0 conn(0x7f8ef4097f80 legacy=0x7f8ef4099400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.622 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 -- 172.21.15.173:0/448475782 shutdown_connections 2021-03-09T09:35:22.622 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 -- 172.21.15.173:0/448475782 >> 172.21.15.173:0/448475782 conn(0x7f8ef406a240 msgr2=0x7f8ef406a650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:22.622 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 -- 172.21.15.173:0/448475782 shutdown_connections 2021-03-09T09:35:22.622 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 1 -- 172.21.15.173:0/448475782 wait complete. 2021-03-09T09:35:22.623 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient: build_initial_monmap 2021-03-09T09:35:22.623 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.593+0000 7f8ef9891700 10 monclient: monmap: 2021-03-09T09:35:22.623 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:22.623 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:22.623 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2021-03-09T09:35:22.624 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2021-03-09T09:35:22.624 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:22.624 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.624 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:22.624 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:22.625 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:22.625 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:22.625 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:22.625 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:22.625 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:22.626 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:22.626 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:22.626 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.626 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 1 Processor -- start 2021-03-09T09:35:22.627 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 1 -- start start 2021-03-09T09:35:22.627 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient: init 2021-03-09T09:35:22.627 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:22.627 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient: _add_conns ranks=[3,5,6,8,4,1,7,2,0] 2021-03-09T09:35:22.627 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient(hunting): picked mon.noname-g con 0x7f8ef4097f80 addr v1:172.21.15.173:6790/0 2021-03-09T09:35:22.628 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient(hunting): picked mon.noname-h con 0x7f8ef4099b10 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:22.628 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient(hunting): picked mon.noname-d con 0x7f8ef4099f70 addr v1:172.21.15.143:6792/0 2021-03-09T09:35:22.628 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 1 -- --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef41154b0 con 0x7f8ef4097f80 2021-03-09T09:35:22.628 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef4115750 con 0x7f8ef4099b10 2021-03-09T09:35:22.628 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 1 -- --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef41159f0 con 0x7f8ef4099f70 2021-03-09T09:35:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient(hunting): authenticate will time out at 723.029204s 2021-03-09T09:35:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8eebfff700 1 --1- >> v1:172.21.15.173:6791/0 conn(0x7f8ef4099b10 0x7f8ef41280e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6791/0 says I am v1:172.21.15.173:56104/0 (socket says 172.21.15.173:56104) 2021-03-09T09:35:22.629 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8eebfff700 1 -- 172.21.15.173:0/426761397 learned_addr learned my addr 172.21.15.173:0/426761397 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef888f700 1 --1- >> v1:172.21.15.173:6790/0 conn(0x7f8ef4097f80 0x7f8ef41279d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6790/0 says I am v1:172.21.15.173:44338/0 (socket says 172.21.15.173:44338) 2021-03-09T09:35:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.3 v1:172.21.15.173:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215861632 0 0) 0x7f8ef41154b0 con 0x7f8ef4097f80 2021-03-09T09:35:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): my global_id is 4116 2021-03-09T09:35:22.630 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.173:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed4003230 con 0x7f8ef4097f80 2021-03-09T09:35:22.631 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344574389 0 0) 0x7f8ef4115750 con 0x7f8ef4099b10 2021-03-09T09:35:22.631 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:22.631 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): my global_id is 4109 2021-03-09T09:35:22.631 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef41154b0 con 0x7f8ef4099b10 2021-03-09T09:35:22.631 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.6 v1:172.21.15.143:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938035700 0 0) 0x7f8ef41159f0 con 0x7f8ef4099f70 2021-03-09T09:35:22.632 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:22.632 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): my global_id is 4137 2021-03-09T09:35:22.632 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.143:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef4115750 con 0x7f8ef4099f70 2021-03-09T09:35:22.632 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4035774773 0 0) 0x7f8ef41154b0 con 0x7f8ef4099b10 2021-03-09T09:35:22.633 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.173:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ef41159f0 con 0x7f8ef4099b10 2021-03-09T09:35:22.633 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f8edc001b40 con 0x7f8ef4099b10 2021-03-09T09:35:22.633 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.633 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2021-03-09T09:35:22.633 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): dump: 2021-03-09T09:35:22.634 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.634 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.634 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.634 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.634 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.635 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.635 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.635 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.635 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.635 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.636 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.636 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.636 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.636 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.636 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.637 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.637 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:22.637 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.3 v1:172.21.15.173:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4043212696 0 0) 0x7f8ed4003230 con 0x7f8ef4097f80 2021-03-09T09:35:22.638 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.173:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ef41154b0 con 0x7f8ef4097f80 2021-03-09T09:35:22.638 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.3 v1:172.21.15.173:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f8ee4004830 con 0x7f8ef4097f80 2021-03-09T09:35:22.638 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.638 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2021-03-09T09:35:22.639 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient(hunting): dump: 2021-03-09T09:35:22.639 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.639 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.639 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.639 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.640 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.640 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.640 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.640 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.640 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.641 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.641 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.641 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.641 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.641 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.642 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.642 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.642 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2894177084 0 0) 0x7f8ef41159f0 con 0x7f8ef4099b10 2021-03-09T09:35:22.642 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 >> v1:172.21.15.143:6792/0 conn(0x7f8ef4099f70 legacy=0x7f8ef4142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.642 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 >> v1:172.21.15.173:6790/0 conn(0x7f8ef4097f80 legacy=0x7f8ef41279d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.643 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient: _finish_hunting 0 2021-03-09T09:35:22.643 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 monclient: found mon.h 2021-03-09T09:35:22.643 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:22.643 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:22.643 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.173:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef4115c90 con 0x7f8ef4099b10 2021-03-09T09:35:22.644 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient: _finish_auth 0 2021-03-09T09:35:22.644 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:22.644 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 5 monclient: authenticate success, global_id 4109 2021-03-09T09:35:22.644 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient: _renew_subs 2021-03-09T09:35:22.644 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:22.645 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.173:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef4096940 con 0x7f8ef4099b10 2021-03-09T09:35:22.645 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient: _renew_subs 2021-03-09T09:35:22.645 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:22.645 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ef9891700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.173:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef4097540 con 0x7f8ef4099b10 2021-03-09T09:35:22.645 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8edc002170 con 0x7f8ef4099b10 2021-03-09T09:35:22.646 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:22.646 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f8edc0026a0 con 0x7f8ef4099b10 2021-03-09T09:35:22.646 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.646 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8eeaffd700 4 set_mon_vals no callback set 2021-03-09T09:35:22.647 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient: got monmap 1 from mon.h (according to old e1) 2021-03-09T09:35:22.647 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 10 monclient: dump: 2021-03-09T09:35:22.647 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.647 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.647 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.648 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.648 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.648 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.648 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.648 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.649 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.649 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.649 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.649 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.649 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.650 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.650 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.650 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.650 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f8edc002ae0 con 0x7f8ef4099b10 2021-03-09T09:35:22.650 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.597+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (unknown 4181032678 0 0) 0x7f8edc006260 con 0x7f8ef4099b10 2021-03-09T09:35:22.651 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.601+0000 7f8ef9891700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:22.651 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.601+0000 7f8ef9891700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:22.651 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.601+0000 7f8ef9891700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:22.651 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.601+0000 7f8ef9891700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.173:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc004d60 con 0x7f8ef4099b10 2021-03-09T09:35:22.652 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.601+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (unknown 1092875540 0 637523725) 0x7f8edc0056d0 con 0x7f8ef4099b10 2021-03-09T09:35:22.652 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.601+0000 7f8ee8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:22.652 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.601+0000 7f8ee8ff9700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:22.686 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.685+0000 7f8ef9891700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "72ccc8f8-8bb0-4230-bfea-c47a8dadaf1c", "id": 0}] 2021-03-09T09:35:22.687 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.685+0000 7f8ef9891700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "72ccc8f8-8bb0-4230-bfea-c47a8dadaf1c", "id": 0}] 2021-03-09T09:35:22.687 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.685+0000 7f8ef9891700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:22.687 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.685+0000 7f8ef9891700 1 -- 172.21.15.173:0/426761397 --> v1:172.21.15.173:6791/0 -- mon_command({"prefix": "osd new", "uuid": "72ccc8f8-8bb0-4230-bfea-c47a8dadaf1c", "id": 0} v 0) v1 -- 0x7f8ecc005010 con 0x7f8ef4099b10 2021-03-09T09:35:22.700 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.697+0000 7f8ee8ff9700 1 -- 172.21.15.173:0/426761397 <== mon.5 v1:172.21.15.173:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "72ccc8f8-8bb0-4230-bfea-c47a8dadaf1c", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2433418821 0 732707515) 0x7f8edc0059d0 con 0x7f8ef4099b10 2021-03-09T09:35:22.700 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.697+0000 7f8ee8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "72ccc8f8-8bb0-4230-bfea-c47a8dadaf1c", "id": 0}] 2021-03-09T09:35:22.700 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.697+0000 7f8ee8ff9700 10 monclient: _finish_command 2 = system:0 2021-03-09T09:35:22.701 INFO:teuthology.orchestra.run.smithi173.stdout:0 2021-03-09T09:35:22.702 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.701+0000 7f8ef9891700 10 monclient: shutdown 2021-03-09T09:35:22.702 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.701+0000 7f8ef9891700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:22.702 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.701+0000 7f8ef9891700 1 -- 172.21.15.173:0/426761397 >> v1:172.21.15.173:6791/0 conn(0x7f8ef4099b10 legacy=0x7f8ef41280e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.702 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.701+0000 7f8ef9891700 1 -- 172.21.15.173:0/426761397 shutdown_connections 2021-03-09T09:35:22.703 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.701+0000 7f8ef9891700 1 -- 172.21.15.173:0/426761397 >> 172.21.15.173:0/426761397 conn(0x7f8ef406a240 msgr2=0x7f8ef412e620 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:22.703 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.701+0000 7f8ef9891700 1 -- 172.21.15.173:0/426761397 shutdown_connections 2021-03-09T09:35:22.703 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.701+0000 7f8ef9891700 1 -- 172.21.15.173:0/426761397 wait complete. 2021-03-09T09:35:22.718 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new dd598087-2308-4d24-9bbc-a6213e89ca6f 1 2021-03-09T09:35:22.889 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.885+0000 7f88d71e2700 10 monclient: get_monmap_and_config 2021-03-09T09:35:22.889 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.885+0000 7f88d71e2700 10 monclient: build_initial_monmap 2021-03-09T09:35:22.889 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.885+0000 7f88d71e2700 10 monclient: monmap: 2021-03-09T09:35:22.890 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:22.890 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.890 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:35:22.891928+0000 2021-03-09T09:35:22.890 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:35:22.891928+0000 2021-03-09T09:35:22.890 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:22.891 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.891 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:22.891 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:22.891 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:22.891 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:22.892 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:22.892 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:22.892 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:22.892 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:22.893 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:22.893 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.901 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 Processor -- start 2021-03-09T09:35:22.901 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 -- start start 2021-03-09T09:35:22.902 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient: init 2021-03-09T09:35:22.902 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:22.902 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient: _add_conns ranks=[6,8,5,3,4,0,2,7,1] 2021-03-09T09:35:22.902 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient(hunting): picked mon.noname-d con 0x7f88d0068210 addr v1:172.21.15.143:6792/0 2021-03-09T09:35:22.902 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient(hunting): picked mon.noname-e con 0x7f88d00685f0 addr v1:172.21.15.143:6793/0 2021-03-09T09:35:22.903 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient(hunting): picked mon.noname-h con 0x7f88d009a700 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:22.903 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00460b0 con 0x7f88d009a700 2021-03-09T09:35:22.903 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 -- --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d0066820 con 0x7f88d0068210 2021-03-09T09:35:22.903 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 -- --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d0066ac0 con 0x7f88d00685f0 2021-03-09T09:35:22.904 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:22.904 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient(hunting): authenticate will time out at 723.321698s 2021-03-09T09:35:22.904 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d69e1700 1 --1- >> v1:172.21.15.173:6791/0 conn(0x7f88d009a700 0x7f88d0130d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6791/0 says I am v1:172.21.15.173:56112/0 (socket says 172.21.15.173:56112) 2021-03-09T09:35:22.904 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d69e1700 1 -- 172.21.15.173:0/3055023334 learned_addr learned my addr 172.21.15.173:0/3055023334 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:22.905 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 -- 172.21.15.173:0/3055023334 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773757157 0 0) 0x7f88d00460b0 con 0x7f88d009a700 2021-03-09T09:35:22.905 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:22.905 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient(hunting): my global_id is 4118 2021-03-09T09:35:22.905 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 -- 172.21.15.173:0/3055023334 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b8003230 con 0x7f88d009a700 2021-03-09T09:35:22.906 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 -- 172.21.15.173:0/3055023334 <== mon.5 v1:172.21.15.173:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 466672358 0 0) 0x7f88b8003230 con 0x7f88d009a700 2021-03-09T09:35:22.906 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 -- 172.21.15.173:0/3055023334 >> v1:172.21.15.143:6793/0 conn(0x7f88d00685f0 legacy=0x7f88d012d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.906 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 -- 172.21.15.173:0/3055023334 >> v1:172.21.15.143:6792/0 conn(0x7f88d0068210 legacy=0x7f88d0128440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.906 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: _finish_hunting 0 2021-03-09T09:35:22.906 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 monclient: found mon.noname-h 2021-03-09T09:35:22.907 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:22.907 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.173:6791/0 2021-03-09T09:35:22.907 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 -- 172.21.15.173:0/3055023334 --> v1:172.21.15.173:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d0066d60 con 0x7f88d009a700 2021-03-09T09:35:22.907 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: _finish_auth 0 2021-03-09T09:35:22.907 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:22.908 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 -- 172.21.15.173:0/3055023334 <== mon.5 v1:172.21.15.173:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f88c4001ba0 con 0x7f88d009a700 2021-03-09T09:35:22.908 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 5 monclient: authenticate success, global_id 4118 2021-03-09T09:35:22.908 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:22.908 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.909 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2021-03-09T09:35:22.909 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: dump: 2021-03-09T09:35:22.909 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.909 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.909 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.910 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.910 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.910 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.910 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.910 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.911 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.911 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.911 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.911 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.912 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.912 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.912 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.912 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.913 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:22.913 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 -- 172.21.15.173:0/3055023334 <== mon.5 v1:172.21.15.173:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88c4001e50 con 0x7f88d009a700 2021-03-09T09:35:22.913 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:22.913 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:22.913 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 1 -- 172.21.15.173:0/3055023334 <== mon.5 v1:172.21.15.173:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f88c4004c70 con 0x7f88d009a700 2021-03-09T09:35:22.914 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 4 set_mon_vals no callback set 2021-03-09T09:35:22.914 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.914 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: got monmap 1 from mon.h (according to old e1) 2021-03-09T09:35:22.915 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d51de700 10 monclient: dump: 2021-03-09T09:35:22.915 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.915 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.916 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.916 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.916 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.916 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.916 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.917 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.917 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.917 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.917 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.918 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.918 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.918 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.918 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.918 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.919 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 10 monclient: shutdown 2021-03-09T09:35:22.919 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:22.919 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 -- 172.21.15.173:0/3055023334 >> v1:172.21.15.173:6791/0 conn(0x7f88d009a700 legacy=0x7f88d0130d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.919 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 -- 172.21.15.173:0/3055023334 shutdown_connections 2021-03-09T09:35:22.920 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 -- 172.21.15.173:0/3055023334 >> 172.21.15.173:0/3055023334 conn(0x7f88d009b7f0 msgr2=0x7f88d009bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:22.920 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 -- 172.21.15.173:0/3055023334 shutdown_connections 2021-03-09T09:35:22.920 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.889+0000 7f88d71e2700 1 -- 172.21.15.173:0/3055023334 wait complete. 2021-03-09T09:35:22.920 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: build_initial_monmap 2021-03-09T09:35:22.920 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: monmap: 2021-03-09T09:35:22.921 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:22.921 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:22.921 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2021-03-09T09:35:22.921 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2021-03-09T09:35:22.921 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:22.922 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.922 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:22.922 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:22.922 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:22.922 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:22.923 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:22.923 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:22.923 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:22.923 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:22.923 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:22.924 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.924 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 1 Processor -- start 2021-03-09T09:35:22.924 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 1 -- start start 2021-03-09T09:35:22.924 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: init 2021-03-09T09:35:22.924 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:22.925 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: _add_conns ranks=[6,1,7,3,2,8,5,0,4] 2021-03-09T09:35:22.925 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient(hunting): picked mon.noname-d con 0x7f88d0068210 addr v1:172.21.15.143:6792/0 2021-03-09T09:35:22.925 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient(hunting): picked mon.noname-f con 0x7f88d00685f0 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:22.925 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient(hunting): picked mon.noname-i con 0x7f88d009a700 addr v1:172.21.15.173:6792/0 2021-03-09T09:35:22.926 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d0065940 con 0x7f88d00685f0 2021-03-09T09:35:22.926 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 1 -- --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d012b740 con 0x7f88d0068210 2021-03-09T09:35:22.926 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 1 -- --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d012b9e0 con 0x7f88d009a700 2021-03-09T09:35:22.926 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:22.926 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient(hunting): authenticate will time out at 723.325522s 2021-03-09T09:35:22.927 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d69e1700 1 --1- >> v1:172.21.15.173:6792/0 conn(0x7f88d009a700 0x7f88d0140e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6792/0 says I am v1:172.21.15.173:37606/0 (socket says 172.21.15.173:37606) 2021-03-09T09:35:22.927 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d69e1700 1 -- 172.21.15.173:0/2719433231 learned_addr learned my addr 172.21.15.173:0/2719433231 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:22.927 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d59df700 1 --1- 172.21.15.173:0/2719433231 >> v1:172.21.15.173:6789/0 conn(0x7f88d00685f0 0x7f88d013d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:60500/0 (socket says 172.21.15.173:60500) 2021-03-09T09:35:22.927 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318042138 0 0) 0x7f88d012b9e0 con 0x7f88d009a700 2021-03-09T09:35:22.928 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:22.928 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): my global_id is 4111 2021-03-09T09:35:22.928 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.173:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b0003230 con 0x7f88d009a700 2021-03-09T09:35:22.928 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084092628 0 0) 0x7f88d0065940 con 0x7f88d00685f0 2021-03-09T09:35:22.928 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:22.929 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): my global_id is 4141 2021-03-09T09:35:22.929 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d012b9e0 con 0x7f88d00685f0 2021-03-09T09:35:22.929 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4135578237 0 0) 0x7f88b0003230 con 0x7f88d009a700 2021-03-09T09:35:22.929 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.173:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88d0065940 con 0x7f88d009a700 2021-03-09T09:35:22.929 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f88c4001b60 con 0x7f88d009a700 2021-03-09T09:35:22.930 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.930 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2021-03-09T09:35:22.930 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): dump: 2021-03-09T09:35:22.930 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.931 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.931 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.931 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.931 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.931 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.932 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.932 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.932 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.932 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.932 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.933 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.933 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.933 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.933 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.933 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.933 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:22.934 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1249054030 0 0) 0x7f88d012b9e0 con 0x7f88d00685f0 2021-03-09T09:35:22.934 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88b0003230 con 0x7f88d00685f0 2021-03-09T09:35:22.934 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f88bc001ac0 con 0x7f88d00685f0 2021-03-09T09:35:22.934 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.935 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2021-03-09T09:35:22.935 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): dump: 2021-03-09T09:35:22.935 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.935 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.935 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.936 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.936 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.936 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.936 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.936 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.937 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.937 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.937 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.937 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.937 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.938 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.938 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.938 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.938 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.6 v1:172.21.15.143:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006622336 0 0) 0x7f88d012b740 con 0x7f88d0068210 2021-03-09T09:35:22.938 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:22.939 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient(hunting): my global_id is 4155 2021-03-09T09:35:22.939 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.143:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d012b9e0 con 0x7f88d0068210 2021-03-09T09:35:22.939 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1473205329 0 0) 0x7f88d0065940 con 0x7f88d009a700 2021-03-09T09:35:22.939 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 >> v1:172.21.15.143:6792/0 conn(0x7f88d0068210 legacy=0x7f88d0130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.940 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 >> v1:172.21.15.173:6789/0 conn(0x7f88d00685f0 legacy=0x7f88d013d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:22.940 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient: _finish_hunting 0 2021-03-09T09:35:22.940 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 monclient: found mon.i 2021-03-09T09:35:22.940 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:22.940 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:22.941 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.173:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d0065ca0 con 0x7f88d009a700 2021-03-09T09:35:22.941 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient: _finish_auth 0 2021-03-09T09:35:22.941 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:22.941 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 5 monclient: authenticate success, global_id 4111 2021-03-09T09:35:22.941 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: _renew_subs 2021-03-09T09:35:22.942 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:22.942 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.173:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d00951e0 con 0x7f88d009a700 2021-03-09T09:35:22.942 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: _renew_subs 2021-03-09T09:35:22.942 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:22.942 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.173:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d0094a70 con 0x7f88d009a700 2021-03-09T09:35:22.943 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88c4001550 con 0x7f88d009a700 2021-03-09T09:35:22.943 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:22.943 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f88c4001fc0 con 0x7f88d009a700 2021-03-09T09:35:22.943 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:22.943 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient: got monmap 1 from mon.i (according to old e1) 2021-03-09T09:35:22.944 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d49dd700 4 set_mon_vals no callback set 2021-03-09T09:35:22.944 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 10 monclient: dump: 2021-03-09T09:35:22.944 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:22.944 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:22.944 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.945 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:22.945 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:22.945 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:22.945 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:22.945 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:22.946 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:22.947 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f88c4005770 con 0x7f88d009a700 2021-03-09T09:35:22.947 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (unknown 957362122 0 0) 0x7f88c4006a40 con 0x7f88d009a700 2021-03-09T09:35:22.948 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:22.948 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:22.948 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:22.948 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.893+0000 7f88d71e2700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.173:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b4004d60 con 0x7f88d009a700 2021-03-09T09:35:22.948 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.897+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (unknown 1092875540 0 637523725) 0x7f88c4005c40 con 0x7f88d009a700 2021-03-09T09:35:22.949 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.897+0000 7f88c27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:22.949 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:22.897+0000 7f88c27fc700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:23.005 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.005+0000 7f88d71e2700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dd598087-2308-4d24-9bbc-a6213e89ca6f", "id": 1}] 2021-03-09T09:35:23.006 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.005+0000 7f88d71e2700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dd598087-2308-4d24-9bbc-a6213e89ca6f", "id": 1}] 2021-03-09T09:35:23.006 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.005+0000 7f88d71e2700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:23.006 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.005+0000 7f88d71e2700 1 -- 172.21.15.173:0/2719433231 --> v1:172.21.15.173:6792/0 -- mon_command({"prefix": "osd new", "uuid": "dd598087-2308-4d24-9bbc-a6213e89ca6f", "id": 1} v 0) v1 -- 0x7f88b4005590 con 0x7f88d009a700 2021-03-09T09:35:23.015 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88c27fc700 1 -- 172.21.15.173:0/2719433231 <== mon.7 v1:172.21.15.173:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dd598087-2308-4d24-9bbc-a6213e89ca6f", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1077466987 0 940483276) 0x7f88c4005f40 con 0x7f88d009a700 2021-03-09T09:35:23.016 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88c27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dd598087-2308-4d24-9bbc-a6213e89ca6f", "id": 1}] 2021-03-09T09:35:23.016 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88c27fc700 10 monclient: _finish_command 2 = system:0 2021-03-09T09:35:23.016 INFO:teuthology.orchestra.run.smithi173.stdout:1 2021-03-09T09:35:23.018 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88d71e2700 10 monclient: shutdown 2021-03-09T09:35:23.018 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88d71e2700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:23.018 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88d71e2700 1 -- 172.21.15.173:0/2719433231 >> v1:172.21.15.173:6792/0 conn(0x7f88d009a700 legacy=0x7f88d0140e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.018 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88d71e2700 1 -- 172.21.15.173:0/2719433231 shutdown_connections 2021-03-09T09:35:23.019 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88d71e2700 1 -- 172.21.15.173:0/2719433231 >> 172.21.15.173:0/2719433231 conn(0x7f88d009b7f0 msgr2=0x7f88d0061750 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:23.019 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88d71e2700 1 -- 172.21.15.173:0/2719433231 shutdown_connections 2021-03-09T09:35:23.019 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.013+0000 7f88d71e2700 1 -- 172.21.15.173:0/2719433231 wait complete. 2021-03-09T09:35:23.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new a8112181-081d-43e4-af54-f8388903c61a 2 2021-03-09T09:35:23.199 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.197+0000 7f508e233700 10 monclient: get_monmap_and_config 2021-03-09T09:35:23.200 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.197+0000 7f508e233700 10 monclient: build_initial_monmap 2021-03-09T09:35:23.200 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.197+0000 7f508e233700 10 monclient: monmap: 2021-03-09T09:35:23.200 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:23.200 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.201 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:35:23.202679+0000 2021-03-09T09:35:23.201 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:35:23.202679+0000 2021-03-09T09:35:23.201 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:23.201 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.201 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:23.202 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:23.202 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:23.202 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:23.202 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:23.203 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:23.203 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:23.203 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:23.203 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:23.203 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.211 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.197+0000 7f508e233700 1 Processor -- start 2021-03-09T09:35:23.211 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.197+0000 7f508e233700 1 -- start start 2021-03-09T09:35:23.211 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.197+0000 7f508e233700 10 monclient: init 2021-03-09T09:35:23.211 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:23.212 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient: _add_conns ranks=[1,2,5,7,8,4,0,3,6] 2021-03-09T09:35:23.212 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): picked mon.noname-f con 0x7f50880981b0 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:23.212 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): picked mon.noname-b con 0x7f5088099bf0 addr v1:172.21.15.143:6790/0 2021-03-09T09:35:23.212 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): picked mon.noname-h con 0x7f508809a070 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:23.212 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50880460b0 con 0x7f50880981b0 2021-03-09T09:35:23.213 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508809b020 con 0x7f5088099bf0 2021-03-09T09:35:23.213 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5088090490 con 0x7f508809a070 2021-03-09T09:35:23.213 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:23.213 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): authenticate will time out at 723.632251s 2021-03-09T09:35:23.214 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50877fe700 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f50880981b0 0x7f50880994e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:60504/0 (socket says 172.21.15.173:60504) 2021-03-09T09:35:23.214 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50877fe700 1 -- 172.21.15.173:0/1958066451 learned_addr learned my addr 172.21.15.173:0/1958066451 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:23.214 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826480603 0 0) 0x7f50880460b0 con 0x7f50880981b0 2021-03-09T09:35:23.214 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.214 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient(hunting): my global_id is 4150 2021-03-09T09:35:23.215 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506c003230 con 0x7f50880981b0 2021-03-09T09:35:23.215 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316804467 0 0) 0x7f5088090490 con 0x7f508809a070 2021-03-09T09:35:23.215 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.215 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient(hunting): my global_id is 4127 2021-03-09T09:35:23.215 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50880460b0 con 0x7f508809a070 2021-03-09T09:35:23.216 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3365917220 0 0) 0x7f506c003230 con 0x7f50880981b0 2021-03-09T09:35:23.216 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 >> v1:172.21.15.173:6791/0 conn(0x7f508809a070 legacy=0x7f5088134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.216 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 >> v1:172.21.15.143:6790/0 conn(0x7f5088099bf0 legacy=0x7f50881309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.216 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: _finish_hunting 0 2021-03-09T09:35:23.217 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 monclient: found mon.noname-f 2021-03-09T09:35:23.217 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:23.217 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.173:6789/0 2021-03-09T09:35:23.217 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5088090730 con 0x7f50880981b0 2021-03-09T09:35:23.217 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: _finish_auth 0 2021-03-09T09:35:23.218 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:23.218 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f5074001820 con 0x7f50880981b0 2021-03-09T09:35:23.218 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.218 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2021-03-09T09:35:23.218 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: dump: 2021-03-09T09:35:23.219 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.219 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.219 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.219 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.219 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.220 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.220 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.220 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.220 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.221 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.221 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.221 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.221 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.221 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.222 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.222 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.222 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 5 monclient: authenticate success, global_id 4150 2021-03-09T09:35:23.222 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:23.222 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5074001a50 con 0x7f50880981b0 2021-03-09T09:35:23.223 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:23.223 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:23.223 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 1 -- 172.21.15.173:0/1958066451 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f50740048f0 con 0x7f50880981b0 2021-03-09T09:35:23.223 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 4 set_mon_vals no callback set 2021-03-09T09:35:23.224 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.224 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: got monmap 1 from mon.f (according to old e1) 2021-03-09T09:35:23.224 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50867fc700 10 monclient: dump: 2021-03-09T09:35:23.224 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.224 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.225 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.225 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.225 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.225 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.225 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.226 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.226 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.226 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.226 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.226 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.227 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.227 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.227 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.227 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.228 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient: shutdown 2021-03-09T09:35:23.228 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:23.228 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- 172.21.15.173:0/1958066451 >> v1:172.21.15.173:6789/0 conn(0x7f50880981b0 legacy=0x7f50880994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.228 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- 172.21.15.173:0/1958066451 shutdown_connections 2021-03-09T09:35:23.228 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- 172.21.15.173:0/1958066451 >> 172.21.15.173:0/1958066451 conn(0x7f508809b780 msgr2=0x7f508809bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:23.229 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- 172.21.15.173:0/1958066451 shutdown_connections 2021-03-09T09:35:23.229 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- 172.21.15.173:0/1958066451 wait complete. 2021-03-09T09:35:23.229 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient: build_initial_monmap 2021-03-09T09:35:23.229 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient: monmap: 2021-03-09T09:35:23.229 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:23.230 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:23.230 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2021-03-09T09:35:23.230 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2021-03-09T09:35:23.230 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:23.231 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.231 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:23.231 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:23.231 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:23.231 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:23.232 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:23.233 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.233 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 Processor -- start 2021-03-09T09:35:23.233 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- start start 2021-03-09T09:35:23.233 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient: init 2021-03-09T09:35:23.233 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:23.234 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient: _add_conns ranks=[5,4,0,2,8,6,3,1,7] 2021-03-09T09:35:23.234 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): picked mon.noname-h con 0x7f50880981b0 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:23.234 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): picked mon.noname-c con 0x7f5088099bf0 addr v1:172.21.15.143:6791/0 2021-03-09T09:35:23.234 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): picked mon.noname-a con 0x7f508809a070 addr v1:172.21.15.143:6789/0 2021-03-09T09:35:23.234 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508811b470 con 0x7f508809a070 2021-03-09T09:35:23.235 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508811b710 con 0x7f5088099bf0 2021-03-09T09:35:23.235 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508811b9b0 con 0x7f50880981b0 2021-03-09T09:35:23.235 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:23.235 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f508e233700 10 monclient(hunting): authenticate will time out at 723.635526s 2021-03-09T09:35:23.235 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50877fe700 1 --1- >> v1:172.21.15.173:6791/0 conn(0x7f50880981b0 0x7f508813ad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6791/0 says I am v1:172.21.15.173:56126/0 (socket says 172.21.15.173:56126) 2021-03-09T09:35:23.236 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50877fe700 1 -- 172.21.15.173:0/3816533828 learned_addr learned my addr 172.21.15.173:0/3816533828 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:23.236 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652185192 0 0) 0x7f508811b9b0 con 0x7f50880981b0 2021-03-09T09:35:23.236 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50737fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.236 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50737fe700 10 monclient(hunting): my global_id is 4136 2021-03-09T09:35:23.236 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.201+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5068003230 con 0x7f50880981b0 2021-03-09T09:35:23.237 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1517046408 0 0) 0x7f5068003230 con 0x7f50880981b0 2021-03-09T09:35:23.237 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 --> v1:172.21.15.173:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f508811b9b0 con 0x7f50880981b0 2021-03-09T09:35:23.237 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f5074004830 con 0x7f50880981b0 2021-03-09T09:35:23.237 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.237 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2021-03-09T09:35:23.238 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient(hunting): dump: 2021-03-09T09:35:23.238 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.238 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.238 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.238 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.239 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.239 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.239 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.239 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.240 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.240 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.240 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.240 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.240 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.241 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.241 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.241 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.241 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:23.241 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3946960849 0 0) 0x7f508811b9b0 con 0x7f50880981b0 2021-03-09T09:35:23.242 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 >> v1:172.21.15.143:6791/0 conn(0x7f5088099bf0 legacy=0x7f508813fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.242 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 >> v1:172.21.15.143:6789/0 conn(0x7f508809a070 legacy=0x7f5088143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.242 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient: _finish_hunting 0 2021-03-09T09:35:23.242 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 monclient: found mon.h 2021-03-09T09:35:23.242 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:23.243 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:23.243 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 --> v1:172.21.15.173:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5088119ae0 con 0x7f50880981b0 2021-03-09T09:35:23.243 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient: _finish_auth 0 2021-03-09T09:35:23.243 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:23.244 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 5 monclient: authenticate success, global_id 4136 2021-03-09T09:35:23.244 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 10 monclient: _renew_subs 2021-03-09T09:35:23.244 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:23.244 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 1 -- 172.21.15.173:0/3816533828 --> v1:172.21.15.173:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508812e7a0 con 0x7f50880981b0 2021-03-09T09:35:23.244 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 10 monclient: _renew_subs 2021-03-09T09:35:23.245 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:23.245 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 1 -- 172.21.15.173:0/3816533828 --> v1:172.21.15.173:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f508813b5e0 con 0x7f50880981b0 2021-03-09T09:35:23.245 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5074001ce0 con 0x7f50880981b0 2021-03-09T09:35:23.245 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:23.245 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f5074001190 con 0x7f50880981b0 2021-03-09T09:35:23.246 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.246 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f5085ffb700 4 set_mon_vals no callback set 2021-03-09T09:35:23.246 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient: got monmap 1 from mon.h (according to old e1) 2021-03-09T09:35:23.246 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 10 monclient: dump: 2021-03-09T09:35:23.246 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.247 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.247 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.247 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.247 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.247 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.248 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.248 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.248 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.248 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.249 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.249 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.249 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.249 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.249 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.249 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.250 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f5074005430 con 0x7f50880981b0 2021-03-09T09:35:23.250 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (unknown 3899268032 0 0) 0x7f50740066c0 con 0x7f50880981b0 2021-03-09T09:35:23.250 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:23.250 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:23.251 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:23.251 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.205+0000 7f508e233700 1 -- 172.21.15.173:0/3816533828 --> v1:172.21.15.173:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5060004d60 con 0x7f50880981b0 2021-03-09T09:35:23.251 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.209+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (unknown 1092875540 0 637523725) 0x7f5074006390 con 0x7f50880981b0 2021-03-09T09:35:23.251 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.209+0000 7f50737fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:23.251 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.209+0000 7f50737fe700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:23.293 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.293+0000 7f508e233700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a8112181-081d-43e4-af54-f8388903c61a", "id": 2}] 2021-03-09T09:35:23.293 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.293+0000 7f508e233700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a8112181-081d-43e4-af54-f8388903c61a", "id": 2}] 2021-03-09T09:35:23.294 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.293+0000 7f508e233700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:23.294 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.293+0000 7f508e233700 1 -- 172.21.15.173:0/3816533828 --> v1:172.21.15.173:6791/0 -- mon_command({"prefix": "osd new", "uuid": "a8112181-081d-43e4-af54-f8388903c61a", "id": 2} v 0) v1 -- 0x7f5060005010 con 0x7f50880981b0 2021-03-09T09:35:23.305 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f50737fe700 1 -- 172.21.15.173:0/3816533828 <== mon.5 v1:172.21.15.173:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a8112181-081d-43e4-af54-f8388903c61a", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1421494437 0 216599125) 0x7f50740069a0 con 0x7f50880981b0 2021-03-09T09:35:23.306 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f50737fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a8112181-081d-43e4-af54-f8388903c61a", "id": 2}] 2021-03-09T09:35:23.306 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f50737fe700 10 monclient: _finish_command 2 = system:0 2021-03-09T09:35:23.306 INFO:teuthology.orchestra.run.smithi173.stdout:2 2021-03-09T09:35:23.307 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f508e233700 10 monclient: shutdown 2021-03-09T09:35:23.307 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f508e233700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:23.308 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f508e233700 1 -- 172.21.15.173:0/3816533828 >> v1:172.21.15.173:6791/0 conn(0x7f50880981b0 legacy=0x7f508813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.308 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f508e233700 1 -- 172.21.15.173:0/3816533828 shutdown_connections 2021-03-09T09:35:23.308 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f508e233700 1 -- 172.21.15.173:0/3816533828 >> 172.21.15.173:0/3816533828 conn(0x7f508809b780 msgr2=0x7f508809e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:23.308 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f508e233700 1 -- 172.21.15.173:0/3816533828 shutdown_connections 2021-03-09T09:35:23.308 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.305+0000 7f508e233700 1 -- 172.21.15.173:0/3816533828 wait complete. 2021-03-09T09:35:23.325 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 23b46e8e-3d64-48b2-be36-1594df839111 3 2021-03-09T09:35:23.496 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.493+0000 7fa1dee49700 10 monclient: get_monmap_and_config 2021-03-09T09:35:23.496 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.493+0000 7fa1dee49700 10 monclient: build_initial_monmap 2021-03-09T09:35:23.496 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.493+0000 7fa1dee49700 10 monclient: monmap: 2021-03-09T09:35:23.497 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:23.497 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.497 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:35:23.499097+0000 2021-03-09T09:35:23.497 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:35:23.499097+0000 2021-03-09T09:35:23.497 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:23.498 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.498 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:23.498 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:23.498 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:23.498 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:23.499 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:23.499 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:23.499 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:23.499 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:23.500 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:23.500 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.510 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.493+0000 7fa1dee49700 1 Processor -- start 2021-03-09T09:35:23.510 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 1 -- start start 2021-03-09T09:35:23.510 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient: init 2021-03-09T09:35:23.510 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:23.511 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient: _add_conns ranks=[0,5,7,1,8,2,6,4,3] 2021-03-09T09:35:23.511 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient(hunting): picked mon.noname-a con 0x7fa1d8096190 addr v1:172.21.15.143:6789/0 2021-03-09T09:35:23.511 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient(hunting): picked mon.noname-h con 0x7fa1d80959b0 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:23.511 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient(hunting): picked mon.noname-i con 0x7fa1d812e4e0 addr v1:172.21.15.173:6792/0 2021-03-09T09:35:23.511 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d80460b0 con 0x7fa1d8096190 2021-03-09T09:35:23.512 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d8090ad0 con 0x7fa1d80959b0 2021-03-09T09:35:23.512 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 1 -- --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d8090d70 con 0x7fa1d812e4e0 2021-03-09T09:35:23.512 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:23.512 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient(hunting): authenticate will time out at 723.929088s 2021-03-09T09:35:23.513 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dd3e6700 1 --1- >> v1:172.21.15.173:6792/0 conn(0x7fa1d812e4e0 0x7fa1d81308d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6792/0 says I am v1:172.21.15.173:37624/0 (socket says 172.21.15.173:37624) 2021-03-09T09:35:23.513 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dd3e6700 1 -- 172.21.15.173:0/2942962819 learned_addr learned my addr 172.21.15.173:0/2942962819 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:23.513 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 <== mon.7 v1:172.21.15.173:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991678947 0 0) 0x7fa1d8090d70 con 0x7fa1d812e4e0 2021-03-09T09:35:23.513 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.513 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient(hunting): my global_id is 4120 2021-03-09T09:35:23.514 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 --> v1:172.21.15.173:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c4003230 con 0x7fa1d812e4e0 2021-03-09T09:35:23.514 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572442191 0 0) 0x7fa1d8090ad0 con 0x7fa1d80959b0 2021-03-09T09:35:23.514 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.514 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient(hunting): my global_id is 4145 2021-03-09T09:35:23.515 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8090d70 con 0x7fa1d80959b0 2021-03-09T09:35:23.515 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519839658 0 0) 0x7fa1d80460b0 con 0x7fa1d8096190 2021-03-09T09:35:23.515 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.515 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient(hunting): my global_id is 4122 2021-03-09T09:35:23.515 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8090ad0 con 0x7fa1d8096190 2021-03-09T09:35:23.516 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 <== mon.7 v1:172.21.15.173:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3988580145 0 0) 0x7fa1c4003230 con 0x7fa1d812e4e0 2021-03-09T09:35:23.516 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 >> v1:172.21.15.173:6791/0 conn(0x7fa1d80959b0 legacy=0x7fa1d812cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.516 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 >> v1:172.21.15.143:6789/0 conn(0x7fa1d8096190 legacy=0x7fa1d8096570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.516 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: _finish_hunting 0 2021-03-09T09:35:23.517 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 monclient: found mon.noname-i 2021-03-09T09:35:23.517 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:23.517 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.173:6792/0 2021-03-09T09:35:23.517 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 --> v1:172.21.15.173:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d8091010 con 0x7fa1d812e4e0 2021-03-09T09:35:23.517 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: _finish_auth 0 2021-03-09T09:35:23.518 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:23.518 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 <== mon.7 v1:172.21.15.173:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fa1d4001c00 con 0x7fa1d812e4e0 2021-03-09T09:35:23.518 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.518 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2021-03-09T09:35:23.519 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: dump: 2021-03-09T09:35:23.519 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.519 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.520 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.520 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.520 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.520 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.521 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.521 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.521 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.521 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.521 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.522 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.522 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.522 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.522 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.522 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.523 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 5 monclient: authenticate success, global_id 4120 2021-03-09T09:35:23.523 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:23.523 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 <== mon.7 v1:172.21.15.173:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1d4001e20 con 0x7fa1d812e4e0 2021-03-09T09:35:23.523 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:23.524 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:23.524 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 1 -- 172.21.15.173:0/2942962819 <== mon.7 v1:172.21.15.173:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fa1d4004c70 con 0x7fa1d812e4e0 2021-03-09T09:35:23.524 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.524 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 4 set_mon_vals no callback set 2021-03-09T09:35:23.524 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: got monmap 1 from mon.i (according to old e1) 2021-03-09T09:35:23.525 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1cf7fe700 10 monclient: dump: 2021-03-09T09:35:23.525 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.525 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.525 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.526 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.526 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.526 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.526 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.526 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.527 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.527 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.527 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.527 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.527 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.528 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.528 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.528 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.528 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 10 monclient: shutdown 2021-03-09T09:35:23.528 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:23.529 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 1 -- 172.21.15.173:0/2942962819 >> v1:172.21.15.173:6792/0 conn(0x7fa1d812e4e0 legacy=0x7fa1d81308d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.529 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 1 -- 172.21.15.173:0/2942962819 shutdown_connections 2021-03-09T09:35:23.529 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 1 -- 172.21.15.173:0/2942962819 >> 172.21.15.173:0/2942962819 conn(0x7fa1d809ba90 msgr2=0x7fa1d8097ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:23.529 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 1 -- 172.21.15.173:0/2942962819 shutdown_connections 2021-03-09T09:35:23.530 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.497+0000 7fa1dee49700 1 -- 172.21.15.173:0/2942962819 wait complete. 2021-03-09T09:35:23.530 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient: build_initial_monmap 2021-03-09T09:35:23.530 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient: monmap: 2021-03-09T09:35:23.530 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:23.530 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:23.531 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2021-03-09T09:35:23.531 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2021-03-09T09:35:23.531 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:23.531 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.532 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:23.532 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:23.532 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:23.532 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:23.532 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:23.533 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:23.533 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:23.533 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:23.533 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:23.533 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.534 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 1 Processor -- start 2021-03-09T09:35:23.534 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 1 -- start start 2021-03-09T09:35:23.534 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient: init 2021-03-09T09:35:23.534 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:23.535 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient: _add_conns ranks=[5,3,7,4,1,8,2,6,0] 2021-03-09T09:35:23.535 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient(hunting): picked mon.noname-h con 0x7fa1d80959b0 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:23.535 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient(hunting): picked mon.noname-g con 0x7fa1d8096190 addr v1:172.21.15.173:6790/0 2021-03-09T09:35:23.535 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient(hunting): picked mon.noname-i con 0x7fa1d812e4e0 addr v1:172.21.15.173:6792/0 2021-03-09T09:35:23.536 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 1 -- --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d8118b80 con 0x7fa1d8096190 2021-03-09T09:35:23.536 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dcbe5700 1 --1- >> v1:172.21.15.173:6791/0 conn(0x7fa1d80959b0 0x7fa1d8115130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6791/0 says I am v1:172.21.15.173:56138/0 (socket says 172.21.15.173:56138) 2021-03-09T09:35:23.536 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dcbe5700 1 -- 172.21.15.173:0/334984117 learned_addr learned my addr 172.21.15.173:0/334984117 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:23.536 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d8118e20 con 0x7fa1d80959b0 2021-03-09T09:35:23.536 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dd3e6700 1 --1- >> v1:172.21.15.173:6792/0 conn(0x7fa1d812e4e0 0x7fa1d81409b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6792/0 says I am v1:172.21.15.173:37630/0 (socket says 172.21.15.173:37630) 2021-03-09T09:35:23.537 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d8116d20 con 0x7fa1d812e4e0 2021-03-09T09:35:23.537 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:23.537 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient(hunting): authenticate will time out at 723.934068s 2021-03-09T09:35:23.537 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810190463 0 0) 0x7fa1d8118b80 con 0x7fa1d8096190 2021-03-09T09:35:23.537 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.538 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): my global_id is 4125 2021-03-09T09:35:23.538 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1b8003230 con 0x7fa1d8096190 2021-03-09T09:35:23.538 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690382515 0 0) 0x7fa1d8118e20 con 0x7fa1d80959b0 2021-03-09T09:35:23.538 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.538 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): my global_id is 4154 2021-03-09T09:35:23.539 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8118b80 con 0x7fa1d80959b0 2021-03-09T09:35:23.539 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.7 v1:172.21.15.173:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511627347 0 0) 0x7fa1d8116d20 con 0x7fa1d812e4e0 2021-03-09T09:35:23.539 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.539 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): my global_id is 4129 2021-03-09T09:35:23.540 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8118e20 con 0x7fa1d812e4e0 2021-03-09T09:35:23.540 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 562094304 0 0) 0x7fa1b8003230 con 0x7fa1d8096190 2021-03-09T09:35:23.540 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1d8116d20 con 0x7fa1d8096190 2021-03-09T09:35:23.540 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fa1c00016a0 con 0x7fa1d8096190 2021-03-09T09:35:23.540 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.541 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2021-03-09T09:35:23.541 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): dump: 2021-03-09T09:35:23.541 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.541 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.541 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.542 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.542 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.542 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.542 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.542 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.543 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.543 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.543 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.543 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.544 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.544 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.544 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.544 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.544 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:23.545 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.7 v1:172.21.15.173:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1462495437 0 0) 0x7fa1d8118e20 con 0x7fa1d812e4e0 2021-03-09T09:35:23.545 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1b8003230 con 0x7fa1d812e4e0 2021-03-09T09:35:23.545 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.5 v1:172.21.15.173:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4214071620 0 0) 0x7fa1d8118b80 con 0x7fa1d80959b0 2021-03-09T09:35:23.547 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1d8118e20 con 0x7fa1d80959b0 2021-03-09T09:35:23.547 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.7 v1:172.21.15.173:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fa1d4004bb0 con 0x7fa1d812e4e0 2021-03-09T09:35:23.547 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.547 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2021-03-09T09:35:23.548 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): dump: 2021-03-09T09:35:23.548 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.548 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.548 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.548 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.549 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.549 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.549 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.549 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.549 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.550 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.550 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.550 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.550 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.551 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.551 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.551 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.551 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.5 v1:172.21.15.173:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fa1c8002b50 con 0x7fa1d80959b0 2021-03-09T09:35:23.551 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.552 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2021-03-09T09:35:23.552 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient(hunting): dump: 2021-03-09T09:35:23.552 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.552 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.553 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.553 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.553 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.553 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.553 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.554 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.554 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.554 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.554 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.554 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.555 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.555 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.555 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.555 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.555 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1553058592 0 0) 0x7fa1d8116d20 con 0x7fa1d8096190 2021-03-09T09:35:23.556 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 >> v1:172.21.15.173:6792/0 conn(0x7fa1d812e4e0 legacy=0x7fa1d81409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.556 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 >> v1:172.21.15.173:6791/0 conn(0x7fa1d80959b0 legacy=0x7fa1d8115130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.556 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient: _finish_hunting 0 2021-03-09T09:35:23.556 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 monclient: found mon.g 2021-03-09T09:35:23.556 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:23.557 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient: _send_mon_message to mon.g at v1:172.21.15.173:6790/0 2021-03-09T09:35:23.557 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d812d7e0 con 0x7fa1d8096190 2021-03-09T09:35:23.557 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient: _finish_auth 0 2021-03-09T09:35:23.557 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:23.558 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 5 monclient: authenticate success, global_id 4125 2021-03-09T09:35:23.558 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient: _renew_subs 2021-03-09T09:35:23.558 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient: _send_mon_message to mon.g at v1:172.21.15.173:6790/0 2021-03-09T09:35:23.558 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d81191e0 con 0x7fa1d8096190 2021-03-09T09:35:23.558 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient: _renew_subs 2021-03-09T09:35:23.559 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 10 monclient: _send_mon_message to mon.g at v1:172.21.15.173:6790/0 2021-03-09T09:35:23.559 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d8117790 con 0x7fa1d8096190 2021-03-09T09:35:23.559 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1c0002030 con 0x7fa1d8096190 2021-03-09T09:35:23.559 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:23.559 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fa1c00024e0 con 0x7fa1d8096190 2021-03-09T09:35:23.560 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.560 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ceffd700 4 set_mon_vals no callback set 2021-03-09T09:35:23.560 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient: got monmap 1 from mon.g (according to old e1) 2021-03-09T09:35:23.560 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 10 monclient: dump: 2021-03-09T09:35:23.560 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.561 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.561 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.561 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.561 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.561 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.562 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.562 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.562 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.562 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.563 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.563 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.563 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.563 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.563 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.564 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.564 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7fa1c00029a0 con 0x7fa1d8096190 2021-03-09T09:35:23.564 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.501+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (unknown 1305119697 0 0) 0x7fa1c0003e50 con 0x7fa1d8096190 2021-03-09T09:35:23.564 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.505+0000 7fa1dee49700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:23.564 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.505+0000 7fa1dee49700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:23.565 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.505+0000 7fa1dee49700 10 monclient: _send_mon_message to mon.g at v1:172.21.15.173:6790/0 2021-03-09T09:35:23.565 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.505+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b0004d60 con 0x7fa1d8096190 2021-03-09T09:35:23.565 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.509+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (unknown 1092875540 0 637523725) 0x7fa1c0003aa0 con 0x7fa1d8096190 2021-03-09T09:35:23.565 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.509+0000 7fa1ccff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:23.565 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.509+0000 7fa1ccff9700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:23.591 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.589+0000 7fa1dee49700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "23b46e8e-3d64-48b2-be36-1594df839111", "id": 3}] 2021-03-09T09:35:23.591 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.589+0000 7fa1dee49700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "23b46e8e-3d64-48b2-be36-1594df839111", "id": 3}] 2021-03-09T09:35:23.591 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.589+0000 7fa1dee49700 10 monclient: _send_mon_message to mon.g at v1:172.21.15.173:6790/0 2021-03-09T09:35:23.591 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.589+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 --> v1:172.21.15.173:6790/0 -- mon_command({"prefix": "osd new", "uuid": "23b46e8e-3d64-48b2-be36-1594df839111", "id": 3} v 0) v1 -- 0x7fa1b0005010 con 0x7fa1d8096190 2021-03-09T09:35:23.601 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1ccff9700 1 -- 172.21.15.173:0/334984117 <== mon.3 v1:172.21.15.173:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "23b46e8e-3d64-48b2-be36-1594df839111", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2356692782 0 525046306) 0x7fa1c0002e60 con 0x7fa1d8096190 2021-03-09T09:35:23.601 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1ccff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "23b46e8e-3d64-48b2-be36-1594df839111", "id": 3}] 2021-03-09T09:35:23.601 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1ccff9700 10 monclient: _finish_command 2 = system:0 2021-03-09T09:35:23.602 INFO:teuthology.orchestra.run.smithi173.stdout:3 2021-03-09T09:35:23.603 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1dee49700 10 monclient: shutdown 2021-03-09T09:35:23.603 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1dee49700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:23.603 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 >> v1:172.21.15.173:6790/0 conn(0x7fa1d8096190 legacy=0x7fa1d8115840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.604 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 shutdown_connections 2021-03-09T09:35:23.604 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 >> 172.21.15.173:0/334984117 conn(0x7fa1d809ba90 msgr2=0x7fa1d80996b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:23.604 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 shutdown_connections 2021-03-09T09:35:23.604 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.601+0000 7fa1dee49700 1 -- 172.21.15.173:0/334984117 wait complete. 2021-03-09T09:35:23.618 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new e50766be-3a46-4493-bb02-6eba34d6437d 4 2021-03-09T09:35:23.790 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient: get_monmap_and_config 2021-03-09T09:35:23.790 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient: build_initial_monmap 2021-03-09T09:35:23.790 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient: monmap: 2021-03-09T09:35:23.791 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:23.791 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.791 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:35:23.792847+0000 2021-03-09T09:35:23.791 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:35:23.792847+0000 2021-03-09T09:35:23.791 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:23.792 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.792 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:23.792 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:23.792 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:23.793 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:23.793 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:23.793 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:23.793 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:23.793 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:23.794 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:23.794 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.802 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 1 Processor -- start 2021-03-09T09:35:23.803 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 1 -- start start 2021-03-09T09:35:23.803 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient: init 2021-03-09T09:35:23.803 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:23.803 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient: _add_conns ranks=[5,4,0,3,1,8,7,2,6] 2021-03-09T09:35:23.804 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient(hunting): picked mon.noname-h con 0x7fe818097f80 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:23.804 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient(hunting): picked mon.noname-c con 0x7fe818099b10 addr v1:172.21.15.143:6791/0 2021-03-09T09:35:23.804 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient(hunting): picked mon.noname-a con 0x7fe818099f70 addr v1:172.21.15.143:6789/0 2021-03-09T09:35:23.804 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180460b0 con 0x7fe818099f70 2021-03-09T09:35:23.804 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 1 -- --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe818090060 con 0x7fe818099b10 2021-03-09T09:35:23.805 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe818090300 con 0x7fe818097f80 2021-03-09T09:35:23.805 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:23.805 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient(hunting): authenticate will time out at 724.222551s 2021-03-09T09:35:23.805 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81e0de700 1 --1- >> v1:172.21.15.173:6791/0 conn(0x7fe818097f80 0x7fe818099400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6791/0 says I am v1:172.21.15.173:56144/0 (socket says 172.21.15.173:56144) 2021-03-09T09:35:23.805 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81e0de700 1 -- 172.21.15.173:0/2095249977 learned_addr learned my addr 172.21.15.173:0/2095249977 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:23.806 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 -- 172.21.15.173:0/2095249977 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506993749 0 0) 0x7fe818090300 con 0x7fe818097f80 2021-03-09T09:35:23.806 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.806 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient(hunting): my global_id is 4163 2021-03-09T09:35:23.806 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 -- 172.21.15.173:0/2095249977 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe800003230 con 0x7fe818097f80 2021-03-09T09:35:23.807 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 -- 172.21.15.173:0/2095249977 <== mon.5 v1:172.21.15.173:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 918618940 0 0) 0x7fe800003230 con 0x7fe818097f80 2021-03-09T09:35:23.807 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 -- 172.21.15.173:0/2095249977 >> v1:172.21.15.143:6791/0 conn(0x7fe818099b10 legacy=0x7fe818130770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.807 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 -- 172.21.15.173:0/2095249977 >> v1:172.21.15.143:6789/0 conn(0x7fe818099f70 legacy=0x7fe818133ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.807 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: _finish_hunting 0 2021-03-09T09:35:23.807 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 monclient: found mon.noname-h 2021-03-09T09:35:23.808 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:23.808 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.173:6791/0 2021-03-09T09:35:23.808 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 -- 172.21.15.173:0/2095249977 --> v1:172.21.15.173:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8180905a0 con 0x7fe818097f80 2021-03-09T09:35:23.808 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: _finish_auth 0 2021-03-09T09:35:23.809 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:23.809 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 5 monclient: authenticate success, global_id 4163 2021-03-09T09:35:23.809 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 -- 172.21.15.173:0/2095249977 <== mon.5 v1:172.21.15.173:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fe808001860 con 0x7fe818097f80 2021-03-09T09:35:23.809 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.809 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2021-03-09T09:35:23.810 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: dump: 2021-03-09T09:35:23.810 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.810 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.810 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.810 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.811 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.811 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.811 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.811 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.811 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.812 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.812 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.812 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.812 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.813 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.813 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.813 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.813 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:23.813 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 -- 172.21.15.173:0/2095249977 <== mon.5 v1:172.21.15.173:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe808001ac0 con 0x7fe818097f80 2021-03-09T09:35:23.814 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:23.814 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:23.814 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 1 -- 172.21.15.173:0/2095249977 <== mon.5 v1:172.21.15.173:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fe8080048f0 con 0x7fe818097f80 2021-03-09T09:35:23.814 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81f0e0700 4 set_mon_vals no callback set 2021-03-09T09:35:23.814 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.815 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: got monmap 1 from mon.h (according to old e1) 2021-03-09T09:35:23.815 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.789+0000 7fe81d0dc700 10 monclient: dump: 2021-03-09T09:35:23.815 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.815 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.815 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.816 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.816 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.816 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.816 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.817 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.817 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.817 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.817 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.817 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.818 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.818 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.818 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.818 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.818 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: shutdown 2021-03-09T09:35:23.819 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:23.819 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- 172.21.15.173:0/2095249977 >> v1:172.21.15.173:6791/0 conn(0x7fe818097f80 legacy=0x7fe818099400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.819 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- 172.21.15.173:0/2095249977 shutdown_connections 2021-03-09T09:35:23.819 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- 172.21.15.173:0/2095249977 >> 172.21.15.173:0/2095249977 conn(0x7fe81806a240 msgr2=0x7fe81806a650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:23.819 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- 172.21.15.173:0/2095249977 shutdown_connections 2021-03-09T09:35:23.820 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- 172.21.15.173:0/2095249977 wait complete. 2021-03-09T09:35:23.820 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: build_initial_monmap 2021-03-09T09:35:23.820 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: monmap: 2021-03-09T09:35:23.820 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:23.820 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:23.821 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2021-03-09T09:35:23.821 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2021-03-09T09:35:23.821 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:23.821 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.821 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:23.822 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:23.822 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:23.822 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:23.822 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:23.823 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:23.823 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:23.823 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:23.823 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:23.823 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.824 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 Processor -- start 2021-03-09T09:35:23.824 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- start start 2021-03-09T09:35:23.824 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: init 2021-03-09T09:35:23.824 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:23.824 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: _add_conns ranks=[1,3,5,2,7,0,4,8,6] 2021-03-09T09:35:23.825 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient(hunting): picked mon.noname-f con 0x7fe818097f80 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:23.825 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient(hunting): picked mon.noname-g con 0x7fe818099b10 addr v1:172.21.15.173:6790/0 2021-03-09T09:35:23.825 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient(hunting): picked mon.noname-h con 0x7fe818099f70 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:23.825 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe818115300 con 0x7fe818097f80 2021-03-09T09:35:23.825 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe818115530 con 0x7fe818099b10 2021-03-09T09:35:23.826 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8181157d0 con 0x7fe818099f70 2021-03-09T09:35:23.826 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:23.826 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient(hunting): authenticate will time out at 724.225626s 2021-03-09T09:35:23.826 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81d8dd700 1 --1- >> v1:172.21.15.173:6790/0 conn(0x7fe818099b10 0x7fe81813fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6790/0 says I am v1:172.21.15.173:44388/0 (socket says 172.21.15.173:44388) 2021-03-09T09:35:23.827 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81d8dd700 1 -- 172.21.15.173:0/942767473 learned_addr learned my addr 172.21.15.173:0/942767473 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:23.827 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81e8df700 1 --1- >> v1:172.21.15.173:6791/0 conn(0x7fe818099f70 0x7fe818143330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6791/0 says I am v1:172.21.15.173:56154/0 (socket says 172.21.15.173:56154) 2021-03-09T09:35:23.827 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150954271 0 0) 0x7fe8181157d0 con 0x7fe818099f70 2021-03-09T09:35:23.827 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.827 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): my global_id is 4172 2021-03-09T09:35:23.828 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f8003230 con 0x7fe818099f70 2021-03-09T09:35:23.828 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097358758 0 0) 0x7fe818115300 con 0x7fe818097f80 2021-03-09T09:35:23.828 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.828 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): my global_id is 4159 2021-03-09T09:35:23.828 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8181157d0 con 0x7fe818097f80 2021-03-09T09:35:23.829 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.3 v1:172.21.15.173:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641851796 0 0) 0x7fe818115530 con 0x7fe818099b10 2021-03-09T09:35:23.829 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:23.829 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): my global_id is 4134 2021-03-09T09:35:23.829 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe818115300 con 0x7fe818099b10 2021-03-09T09:35:23.830 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3833070679 0 0) 0x7fe8181157d0 con 0x7fe818097f80 2021-03-09T09:35:23.830 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe818115530 con 0x7fe818097f80 2021-03-09T09:35:23.830 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fe808004830 con 0x7fe818097f80 2021-03-09T09:35:23.830 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.830 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2021-03-09T09:35:23.831 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): dump: 2021-03-09T09:35:23.831 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.831 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.831 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.831 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.832 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.832 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.832 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.832 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.832 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.833 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.833 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.833 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.833 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.833 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.834 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.834 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.834 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:23.834 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.3 v1:172.21.15.173:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3405213504 0 0) 0x7fe818115300 con 0x7fe818099b10 2021-03-09T09:35:23.835 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8181157d0 con 0x7fe818099b10 2021-03-09T09:35:23.835 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.5 v1:172.21.15.173:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1171550160 0 0) 0x7fe7f8003230 con 0x7fe818099f70 2021-03-09T09:35:23.835 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe818115300 con 0x7fe818099f70 2021-03-09T09:35:23.835 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.5 v1:172.21.15.173:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fe814001d80 con 0x7fe818099f70 2021-03-09T09:35:23.835 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.836 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2021-03-09T09:35:23.836 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): dump: 2021-03-09T09:35:23.836 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.836 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.836 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.837 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.837 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.837 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.837 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.837 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.838 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.838 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.838 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.838 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.838 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.839 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.839 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.839 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.839 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.3 v1:172.21.15.173:6790/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fe80c001ac0 con 0x7fe818099b10 2021-03-09T09:35:23.839 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.840 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2021-03-09T09:35:23.840 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient(hunting): dump: 2021-03-09T09:35:23.840 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.840 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.841 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.841 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.841 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.841 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.841 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.841 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.842 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.842 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.842 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.842 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.843 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.843 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.843 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.843 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.843 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 74547613 0 0) 0x7fe818115530 con 0x7fe818097f80 2021-03-09T09:35:23.844 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 >> v1:172.21.15.173:6791/0 conn(0x7fe818099f70 legacy=0x7fe818143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.844 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 >> v1:172.21.15.173:6790/0 conn(0x7fe818099b10 legacy=0x7fe81813fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.844 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient: _finish_hunting 0 2021-03-09T09:35:23.844 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 monclient: found mon.f 2021-03-09T09:35:23.844 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:23.845 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:23.845 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe818115a70 con 0x7fe818097f80 2021-03-09T09:35:23.845 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient: _finish_auth 0 2021-03-09T09:35:23.845 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:23.845 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 5 monclient: authenticate success, global_id 4159 2021-03-09T09:35:23.846 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: _renew_subs 2021-03-09T09:35:23.846 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:23.846 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe818096880 con 0x7fe818097f80 2021-03-09T09:35:23.846 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: _renew_subs 2021-03-09T09:35:23.847 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:23.847 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81f0e0700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81813b5e0 con 0x7fe818097f80 2021-03-09T09:35:23.847 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe808000fa0 con 0x7fe818097f80 2021-03-09T09:35:23.847 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:23.847 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe81c8db700 4 set_mon_vals no callback set 2021-03-09T09:35:23.848 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7fe808001110 con 0x7fe818097f80 2021-03-09T09:35:23.848 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:23.848 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient: got monmap 1 from mon.f (according to old e1) 2021-03-09T09:35:23.848 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 10 monclient: dump: 2021-03-09T09:35:23.848 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:23.849 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:23.849 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.849 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:23.849 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:23.849 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:23.850 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:23.850 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:23.850 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:23.850 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:23.851 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:23.851 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:23.851 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:23.851 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:23.851 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:23.852 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:23.852 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7fe808005480 con 0x7fe818097f80 2021-03-09T09:35:23.852 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.793+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (unknown 4100697614 0 0) 0x7fe808006690 con 0x7fe818097f80 2021-03-09T09:35:23.852 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.797+0000 7fe81f0e0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:23.852 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.797+0000 7fe81f0e0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:23.853 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.797+0000 7fe81f0e0700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:23.853 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.797+0000 7fe81f0e0700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc004d60 con 0x7fe818097f80 2021-03-09T09:35:23.853 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.797+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (unknown 1092875540 0 637523725) 0x7fe808006230 con 0x7fe818097f80 2021-03-09T09:35:23.853 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.797+0000 7fe8067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:23.853 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.797+0000 7fe8067fc700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:23.884 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.881+0000 7fe81f0e0700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e50766be-3a46-4493-bb02-6eba34d6437d", "id": 4}] 2021-03-09T09:35:23.884 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.881+0000 7fe81f0e0700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e50766be-3a46-4493-bb02-6eba34d6437d", "id": 4}] 2021-03-09T09:35:23.884 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.881+0000 7fe81f0e0700 10 monclient: _send_mon_message to mon.f at v1:172.21.15.173:6789/0 2021-03-09T09:35:23.885 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.881+0000 7fe81f0e0700 1 -- 172.21.15.173:0/942767473 --> v1:172.21.15.173:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e50766be-3a46-4493-bb02-6eba34d6437d", "id": 4} v 0) v1 -- 0x7fe7fc005590 con 0x7fe818097f80 2021-03-09T09:35:23.894 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe8067fc700 1 -- 172.21.15.173:0/942767473 <== mon.1 v1:172.21.15.173:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e50766be-3a46-4493-bb02-6eba34d6437d", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 760426351 0 1697012583) 0x7fe8080057b0 con 0x7fe818097f80 2021-03-09T09:35:23.895 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe8067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e50766be-3a46-4493-bb02-6eba34d6437d", "id": 4}] 2021-03-09T09:35:23.895 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe8067fc700 10 monclient: _finish_command 2 = system:0 2021-03-09T09:35:23.895 INFO:teuthology.orchestra.run.smithi173.stdout:4 2021-03-09T09:35:23.896 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe81f0e0700 10 monclient: shutdown 2021-03-09T09:35:23.896 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe81f0e0700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:23.897 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe81f0e0700 1 -- 172.21.15.173:0/942767473 >> v1:172.21.15.173:6789/0 conn(0x7fe818097f80 legacy=0x7fe81813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:23.897 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe81f0e0700 1 -- 172.21.15.173:0/942767473 shutdown_connections 2021-03-09T09:35:23.897 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe81f0e0700 1 -- 172.21.15.173:0/942767473 >> 172.21.15.173:0/942767473 conn(0x7fe81806a240 msgr2=0x7fe81812e600 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:23.897 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe81f0e0700 1 -- 172.21.15.173:0/942767473 shutdown_connections 2021-03-09T09:35:23.898 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:23.893+0000 7fe81f0e0700 1 -- 172.21.15.173:0/942767473 wait complete. 2021-03-09T09:35:23.915 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 1169fffe-55ed-4cf1-b718-b9bf3d9ffe2c 5 2021-03-09T09:35:24.086 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient: get_monmap_and_config 2021-03-09T09:35:24.086 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient: build_initial_monmap 2021-03-09T09:35:24.087 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient: monmap: 2021-03-09T09:35:24.087 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:24.087 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:24.087 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:35:24.089343+0000 2021-03-09T09:35:24.088 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:35:24.089343+0000 2021-03-09T09:35:24.088 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:24.088 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:24.089 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:24.089 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:24.089 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:24.089 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:24.090 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:24.091 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:24.100 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 1 Processor -- start 2021-03-09T09:35:24.100 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 1 -- start start 2021-03-09T09:35:24.100 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient: init 2021-03-09T09:35:24.101 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:24.101 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient: _add_conns ranks=[1,2,4,8,7,0,3,5,6] 2021-03-09T09:35:24.101 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient(hunting): picked mon.noname-f con 0x7f0a04099dd0 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:24.101 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient(hunting): picked mon.noname-b con 0x7f0a0409a1b0 addr v1:172.21.15.143:6790/0 2021-03-09T09:35:24.102 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient(hunting): picked mon.noname-c con 0x7f0a0409a630 addr v1:172.21.15.143:6791/0 2021-03-09T09:35:24.102 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a040460b0 con 0x7f0a04099dd0 2021-03-09T09:35:24.102 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a04096b20 con 0x7f0a0409a1b0 2021-03-09T09:35:24.102 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 1 -- --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a04096dc0 con 0x7f0a0409a630 2021-03-09T09:35:24.103 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:24.103 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient(hunting): authenticate will time out at 724.518645s 2021-03-09T09:35:24.103 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a08d17700 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f0a04099dd0 0x7f0a040995b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:60540/0 (socket says 172.21.15.173:60540) 2021-03-09T09:35:24.103 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a08d17700 1 -- 172.21.15.173:0/1058139826 learned_addr learned my addr 172.21.15.173:0/1058139826 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:24.103 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 -- 172.21.15.173:0/1058139826 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590006657 0 0) 0x7f0a040460b0 con 0x7f0a04099dd0 2021-03-09T09:35:24.104 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:24.104 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient(hunting): my global_id is 4168 2021-03-09T09:35:24.104 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 -- 172.21.15.173:0/1058139826 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f0003230 con 0x7f0a04099dd0 2021-03-09T09:35:24.104 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 -- 172.21.15.173:0/1058139826 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1467404714 0 0) 0x7f09f0003230 con 0x7f0a04099dd0 2021-03-09T09:35:24.104 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 -- 172.21.15.173:0/1058139826 >> v1:172.21.15.143:6791/0 conn(0x7f0a0409a630 legacy=0x7f0a04134090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:24.105 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 -- 172.21.15.173:0/1058139826 >> v1:172.21.15.143:6790/0 conn(0x7f0a0409a1b0 legacy=0x7f0a04130960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:24.105 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: _finish_hunting 0 2021-03-09T09:35:24.105 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 monclient: found mon.noname-f 2021-03-09T09:35:24.105 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:24.106 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: _send_mon_message to mon.noname-f at v1:172.21.15.173:6789/0 2021-03-09T09:35:24.106 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 -- 172.21.15.173:0/1058139826 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a04097060 con 0x7f0a04099dd0 2021-03-09T09:35:24.106 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: _finish_auth 0 2021-03-09T09:35:24.106 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:24.106 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 5 monclient: authenticate success, global_id 4168 2021-03-09T09:35:24.107 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 -- 172.21.15.173:0/1058139826 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f09f4001820 con 0x7f0a04099dd0 2021-03-09T09:35:24.107 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:24.107 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:24.107 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2021-03-09T09:35:24.107 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: dump: 2021-03-09T09:35:24.108 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:24.108 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:24.108 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:24.108 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:24.109 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:24.109 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:24.109 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:24.109 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:24.109 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:24.110 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:24.110 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:24.110 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:24.110 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:24.111 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:24.111 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:24.111 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:24.111 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:24.111 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 -- 172.21.15.173:0/1058139826 <== mon.1 v1:172.21.15.173:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09f4001a40 con 0x7f0a04099dd0 2021-03-09T09:35:24.112 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:24.112 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 1 -- 172.21.15.173:0/1058139826 <== mon.1 v1:172.21.15.173:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f09f40048f0 con 0x7f0a04099dd0 2021-03-09T09:35:24.112 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:24.112 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: got monmap 1 from mon.f (according to old e1) 2021-03-09T09:35:24.113 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f09fb7fe700 10 monclient: dump: 2021-03-09T09:35:24.113 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:24.113 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:24.113 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:24.113 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:24.114 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:24.114 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:24.114 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:24.114 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:24.115 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:24.115 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:24.115 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:24.115 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:24.115 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:24.116 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:24.116 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:24.116 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:24.116 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:24.116 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.085+0000 7f0a0af7b700 4 set_mon_vals no callback set 2021-03-09T09:35:24.117 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient: shutdown 2021-03-09T09:35:24.117 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:24.117 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1058139826 >> v1:172.21.15.173:6789/0 conn(0x7f0a04099dd0 legacy=0x7f0a040995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:24.117 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1058139826 shutdown_connections 2021-03-09T09:35:24.118 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1058139826 >> 172.21.15.173:0/1058139826 conn(0x7f0a0409b850 msgr2=0x7f0a0409bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:24.118 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1058139826 shutdown_connections 2021-03-09T09:35:24.118 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1058139826 wait complete. 2021-03-09T09:35:24.118 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient: build_initial_monmap 2021-03-09T09:35:24.118 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient: monmap: 2021-03-09T09:35:24.119 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 0 2021-03-09T09:35:24.119 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:24.119 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 0.000000 2021-03-09T09:35:24.119 INFO:teuthology.orchestra.run.smithi173.stderr:created 0.000000 2021-03-09T09:35:24.119 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:24.120 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:24.120 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:24.120 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:24.120 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:24.120 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:24.121 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:24.121 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:24.121 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:24.121 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:24.121 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:24.122 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:24.122 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 Processor -- start 2021-03-09T09:35:24.122 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- start start 2021-03-09T09:35:24.122 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient: init 2021-03-09T09:35:24.123 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:24.123 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient: _add_conns ranks=[2,7,0,6,5,1,3,4,8] 2021-03-09T09:35:24.123 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient(hunting): picked mon.noname-b con 0x7f0a04099dd0 addr v1:172.21.15.143:6790/0 2021-03-09T09:35:24.124 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient(hunting): picked mon.noname-i con 0x7f0a0409a1b0 addr v1:172.21.15.173:6792/0 2021-03-09T09:35:24.124 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient(hunting): picked mon.noname-a con 0x7f0a0409a630 addr v1:172.21.15.143:6789/0 2021-03-09T09:35:24.124 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0411ab30 con 0x7f0a0409a630 2021-03-09T09:35:24.124 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0411ad60 con 0x7f0a04099dd0 2021-03-09T09:35:24.125 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0411b000 con 0x7f0a0409a1b0 2021-03-09T09:35:24.125 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:24.126 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient(hunting): authenticate will time out at 724.522904s 2021-03-09T09:35:24.126 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09fbfff700 1 --1- >> v1:172.21.15.173:6792/0 conn(0x7f0a0409a1b0 0x7f0a0413fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6792/0 says I am v1:172.21.15.173:37652/0 (socket says 172.21.15.173:37652) 2021-03-09T09:35:24.127 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09fbfff700 1 -- 172.21.15.173:0/1804890973 learned_addr learned my addr 172.21.15.173:0/1804890973 (peer_addr_for_me v1:172.21.15.173:0/0) 2021-03-09T09:35:24.127 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144921289 0 0) 0x7f0a0411b000 con 0x7f0a0409a1b0 2021-03-09T09:35:24.127 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:24.127 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): my global_id is 4138 2021-03-09T09:35:24.127 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 --> v1:172.21.15.173:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e4003230 con 0x7f0a0409a1b0 2021-03-09T09:35:24.128 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459661835 0 0) 0x7f0a0411ad60 con 0x7f0a04099dd0 2021-03-09T09:35:24.128 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:24.128 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): my global_id is 4133 2021-03-09T09:35:24.128 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0411b000 con 0x7f0a04099dd0 2021-03-09T09:35:24.128 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2571705568 0 0) 0x7f09e4003230 con 0x7f0a0409a1b0 2021-03-09T09:35:24.129 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 --> v1:172.21.15.173:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a0411ad60 con 0x7f0a0409a1b0 2021-03-09T09:35:24.129 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f09ec001ab0 con 0x7f0a0409a1b0 2021-03-09T09:35:24.129 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:24.129 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2021-03-09T09:35:24.130 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): dump: 2021-03-09T09:35:24.130 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:24.130 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:24.130 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:24.130 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:24.131 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:24.131 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:24.131 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:24.131 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:24.131 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:24.132 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:24.132 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:24.132 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:24.132 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:24.132 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:24.133 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:24.133 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:24.133 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:24.133 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815275840 0 0) 0x7f0a0411ab30 con 0x7f0a0409a630 2021-03-09T09:35:24.133 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:24.134 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient(hunting): my global_id is 4140 2021-03-09T09:35:24.134 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e4003230 con 0x7f0a0409a630 2021-03-09T09:35:24.134 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2632810318 0 0) 0x7f0a0411ad60 con 0x7f0a0409a1b0 2021-03-09T09:35:24.134 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 >> v1:172.21.15.143:6790/0 conn(0x7f0a04099dd0 legacy=0x7f0a0412d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:24.134 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 >> v1:172.21.15.143:6789/0 conn(0x7f0a0409a630 legacy=0x7f0a041434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:24.135 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient: _finish_hunting 0 2021-03-09T09:35:24.135 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 monclient: found mon.i 2021-03-09T09:35:24.135 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:24.135 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:24.135 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 --> v1:172.21.15.173:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a040959f0 con 0x7f0a0409a1b0 2021-03-09T09:35:24.136 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 10 monclient: _finish_auth 0 2021-03-09T09:35:24.136 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f09f8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:24.136 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 5 monclient: authenticate success, global_id 4138 2021-03-09T09:35:24.136 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient: _renew_subs 2021-03-09T09:35:24.136 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:24.137 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1804890973 --> v1:172.21.15.173:6792/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a04118e90 con 0x7f0a0409a1b0 2021-03-09T09:35:24.137 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.089+0000 7f0a0af7b700 10 monclient: _renew_subs 2021-03-09T09:35:24.137 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f0a0af7b700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:24.137 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1804890973 --> v1:172.21.15.173:6792/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0412de70 con 0x7f0a0409a1b0 2021-03-09T09:35:24.138 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f0a0af7b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:24.138 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f0a0af7b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:24.138 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f0a0af7b700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:24.138 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09ec002270 con 0x7f0a0409a1b0 2021-03-09T09:35:24.138 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1804890973 --> v1:172.21.15.173:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a04119970 con 0x7f0a0409a1b0 2021-03-09T09:35:24.139 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:24.139 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f09ec002720 con 0x7f0a0409a1b0 2021-03-09T09:35:24.139 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:24.139 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 10 monclient: got monmap 1 from mon.i (according to old e1) 2021-03-09T09:35:24.139 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09fa7fc700 4 set_mon_vals no callback set 2021-03-09T09:35:24.140 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 10 monclient: dump: 2021-03-09T09:35:24.140 INFO:teuthology.orchestra.run.smithi173.stderr:epoch 1 2021-03-09T09:35:24.140 INFO:teuthology.orchestra.run.smithi173.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:24.140 INFO:teuthology.orchestra.run.smithi173.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:24.140 INFO:teuthology.orchestra.run.smithi173.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:24.141 INFO:teuthology.orchestra.run.smithi173.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:24.141 INFO:teuthology.orchestra.run.smithi173.stderr:election_strategy: 1 2021-03-09T09:35:24.141 INFO:teuthology.orchestra.run.smithi173.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:24.141 INFO:teuthology.orchestra.run.smithi173.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:24.141 INFO:teuthology.orchestra.run.smithi173.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:24.142 INFO:teuthology.orchestra.run.smithi173.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:24.142 INFO:teuthology.orchestra.run.smithi173.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:24.142 INFO:teuthology.orchestra.run.smithi173.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:24.142 INFO:teuthology.orchestra.run.smithi173.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:24.142 INFO:teuthology.orchestra.run.smithi173.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:24.143 INFO:teuthology.orchestra.run.smithi173.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:24.143 INFO:teuthology.orchestra.run.smithi173.stderr: 2021-03-09T09:35:24.143 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f09ec002b60 con 0x7f0a0409a1b0 2021-03-09T09:35:24.143 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (unknown 1048813180 0 0) 0x7f09ec006470 con 0x7f0a0409a1b0 2021-03-09T09:35:24.143 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (unknown 1092875540 0 637523725) 0x7f09ec005fe0 con 0x7f0a0409a1b0 2021-03-09T09:35:24.144 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:24.144 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.093+0000 7f09f8ff9700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:24.177 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.177+0000 7f0a0af7b700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1169fffe-55ed-4cf1-b718-b9bf3d9ffe2c", "id": 5}] 2021-03-09T09:35:24.178 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.177+0000 7f0a0af7b700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1169fffe-55ed-4cf1-b718-b9bf3d9ffe2c", "id": 5}] 2021-03-09T09:35:24.178 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.177+0000 7f0a0af7b700 10 monclient: _send_mon_message to mon.i at v1:172.21.15.173:6792/0 2021-03-09T09:35:24.178 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.177+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1804890973 --> v1:172.21.15.173:6792/0 -- mon_command({"prefix": "osd new", "uuid": "1169fffe-55ed-4cf1-b718-b9bf3d9ffe2c", "id": 5} v 0) v1 -- 0x7f0a0405d1a0 con 0x7f0a0409a1b0 2021-03-09T09:35:24.186 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f09f8ff9700 1 -- 172.21.15.173:0/1804890973 <== mon.7 v1:172.21.15.173:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1169fffe-55ed-4cf1-b718-b9bf3d9ffe2c", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1891649978 0 1988412176) 0x7f09ec006200 con 0x7f0a0409a1b0 2021-03-09T09:35:24.187 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f09f8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1169fffe-55ed-4cf1-b718-b9bf3d9ffe2c", "id": 5}] 2021-03-09T09:35:24.187 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f09f8ff9700 10 monclient: _finish_command 2 = system:0 2021-03-09T09:35:24.187 INFO:teuthology.orchestra.run.smithi173.stdout:5 2021-03-09T09:35:24.189 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f0a0af7b700 10 monclient: shutdown 2021-03-09T09:35:24.189 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f0a0af7b700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:24.189 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1804890973 >> v1:172.21.15.173:6792/0 conn(0x7f0a0409a1b0 legacy=0x7f0a0413fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:24.189 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1804890973 shutdown_connections 2021-03-09T09:35:24.190 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1804890973 >> 172.21.15.173:0/1804890973 conn(0x7f0a0409b850 msgr2=0x7f0a0412e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:24.190 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1804890973 shutdown_connections 2021-03-09T09:35:24.190 INFO:teuthology.orchestra.run.smithi173.stderr:2021-03-09T09:35:24.185+0000 7f0a0af7b700 1 -- 172.21.15.173:0/1804890973 wait complete. 2021-03-09T09:35:24.204 INFO:tasks.ceph.osd.0:Restarting daemon 2021-03-09T09:35:24.204 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-03-09T09:35:24.206 INFO:tasks.ceph.osd.0:Started 2021-03-09T09:35:24.207 INFO:tasks.ceph.osd.1:Restarting daemon 2021-03-09T09:35:24.207 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-03-09T09:35:24.209 INFO:tasks.ceph.osd.1:Started 2021-03-09T09:35:24.210 INFO:tasks.ceph.osd.2:Restarting daemon 2021-03-09T09:35:24.210 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-03-09T09:35:24.212 INFO:tasks.ceph.osd.2:Started 2021-03-09T09:35:24.212 INFO:tasks.ceph.osd.3:Restarting daemon 2021-03-09T09:35:24.213 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-03-09T09:35:24.215 INFO:tasks.ceph.osd.3:Started 2021-03-09T09:35:24.215 INFO:tasks.ceph.osd.4:Restarting daemon 2021-03-09T09:35:24.216 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2021-03-09T09:35:24.217 INFO:tasks.ceph.osd.4:Started 2021-03-09T09:35:24.218 INFO:tasks.ceph.osd.5:Restarting daemon 2021-03-09T09:35:24.218 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2021-03-09T09:35:24.220 INFO:tasks.ceph.osd.5:Started 2021-03-09T09:35:24.221 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-09T09:35:24.826 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T09:35:24.894 INFO:tasks.ceph.osd.3.smithi173.stderr:2021-03-09T09:35:24.893+0000 7f02a938ff00 -1 Falling back to public interface 2021-03-09T09:35:24.894 INFO:tasks.ceph.osd.0.smithi143.stderr:2021-03-09T09:35:24.894+0000 7f28c0f50f00 -1 Falling back to public interface 2021-03-09T09:35:24.895 INFO:tasks.ceph.osd.1.smithi143.stderr:2021-03-09T09:35:24.894+0000 7f471cef6f00 -1 Falling back to public interface 2021-03-09T09:35:24.896 INFO:tasks.ceph.osd.2.smithi143.stderr:2021-03-09T09:35:24.894+0000 7ff44474df00 -1 Falling back to public interface 2021-03-09T09:35:24.897 INFO:tasks.ceph.osd.4.smithi173.stderr:2021-03-09T09:35:24.897+0000 7fa740c4df00 -1 Falling back to public interface 2021-03-09T09:35:24.910 INFO:tasks.ceph.osd.5.smithi173.stderr:2021-03-09T09:35:24.909+0000 7f3b102f0f00 -1 Falling back to public interface 2021-03-09T09:35:25.003 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient: get_monmap_and_config 2021-03-09T09:35:25.004 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient: build_initial_monmap 2021-03-09T09:35:25.004 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient: monmap: 2021-03-09T09:35:25.004 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2021-03-09T09:35:25.005 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:25.005 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:35:25.006495+0000 2021-03-09T09:35:25.005 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:35:25.006495+0000 2021-03-09T09:35:25.006 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:25.006 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:25.006 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:25.007 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:25.007 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:25.007 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:25.008 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:25.008 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:25.008 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:25.010 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:25.010 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:25.011 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:25.025 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 1 Processor -- start 2021-03-09T09:35:25.025 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 1 -- start start 2021-03-09T09:35:25.025 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient: init 2021-03-09T09:35:25.026 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:25.026 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient: _add_conns ranks=[5,1,0,6,8,4,2,7,3] 2021-03-09T09:35:25.026 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient(hunting): picked mon.noname-h con 0x7f1f800bb210 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:25.026 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient(hunting): picked mon.noname-f con 0x7f1f800bb5f0 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:25.027 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient(hunting): picked mon.noname-a con 0x7f1f800c17c0 addr v1:172.21.15.143:6789/0 2021-03-09T09:35:25.027 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f80003dc0 con 0x7f1f800c17c0 2021-03-09T09:35:25.027 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f80025070 con 0x7f1f800bb5f0 2021-03-09T09:35:25.028 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f80026220 con 0x7f1f800bb210 2021-03-09T09:35:25.028 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:25.028 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f87fff700 10 monclient(hunting): authenticate will time out at 714.411450s 2021-03-09T09:35:25.028 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f877fe700 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1f800c17c0 0x7f1f800c3bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:57104/0 (socket says 172.21.15.143:57104) 2021-03-09T09:35:25.029 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.002+0000 7f1f877fe700 1 -- 172.21.15.143:0/2398214479 learned_addr learned my addr 172.21.15.143:0/2398214479 (peer_addr_for_me v1:172.21.15.143:0/0) 2021-03-09T09:35:25.029 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379669895 0 0) 0x7f1f80003dc0 con 0x7f1f800c17c0 2021-03-09T09:35:25.029 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:25.030 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient(hunting): my global_id is 4149 2021-03-09T09:35:25.030 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c003230 con 0x7f1f800c17c0 2021-03-09T09:35:25.031 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905810587 0 0) 0x7f1f80025070 con 0x7f1f800bb5f0 2021-03-09T09:35:25.031 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:25.031 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient(hunting): my global_id is 4177 2021-03-09T09:35:25.032 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f80003dc0 con 0x7f1f800bb5f0 2021-03-09T09:35:25.032 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869670491 0 0) 0x7f1f80026220 con 0x7f1f800bb210 2021-03-09T09:35:25.032 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:25.033 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient(hunting): my global_id is 4217 2021-03-09T09:35:25.034 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f80025070 con 0x7f1f800bb210 2021-03-09T09:35:25.034 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3781164503 0 0) 0x7f1f6c003230 con 0x7f1f800c17c0 2021-03-09T09:35:25.035 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 >> v1:172.21.15.173:6791/0 conn(0x7f1f800bb210 legacy=0x7f1f800280f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:25.035 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 >> v1:172.21.15.173:6789/0 conn(0x7f1f800bb5f0 legacy=0x7f1f800bba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:25.035 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: _finish_hunting 0 2021-03-09T09:35:25.036 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 monclient: found mon.noname-a 2021-03-09T09:35:25.036 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:25.037 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2021-03-09T09:35:25.037 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f800276a0 con 0x7f1f800c17c0 2021-03-09T09:35:25.037 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: _finish_auth 0 2021-03-09T09:35:25.038 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:25.038 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 5 monclient: authenticate success, global_id 4149 2021-03-09T09:35:25.038 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:25.039 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f1f78001ba0 con 0x7f1f800c17c0 2021-03-09T09:35:25.039 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:25.040 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-03-09T09:35:25.040 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: dump: 2021-03-09T09:35:25.040 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2021-03-09T09:35:25.041 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:25.041 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:25.041 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:25.041 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:25.042 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:25.042 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:25.043 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:25.043 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:25.043 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:25.044 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:25.044 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:25.044 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:25.044 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:25.045 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:25.045 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:25.045 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:25.046 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f78001e10 con 0x7f1f800c17c0 2021-03-09T09:35:25.046 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:25.046 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:25.046 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 4 set_mon_vals no callback set 2021-03-09T09:35:25.047 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 1 -- 172.21.15.143:0/2398214479 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f1f78004c70 con 0x7f1f800c17c0 2021-03-09T09:35:25.047 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:25.048 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-03-09T09:35:25.048 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f85ffb700 10 monclient: dump: 2021-03-09T09:35:25.048 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2021-03-09T09:35:25.049 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:25.049 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:25.050 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:25.050 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:25.050 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:25.051 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:25.051 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:25.051 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:25.052 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:25.052 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:25.052 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:25.053 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:25.053 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:25.053 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:25.054 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:25.054 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 10 monclient: shutdown 2021-03-09T09:35:25.054 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:25.055 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 1 -- 172.21.15.143:0/2398214479 >> v1:172.21.15.143:6789/0 conn(0x7f1f800c17c0 legacy=0x7f1f800c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:25.055 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 1 -- 172.21.15.143:0/2398214479 shutdown_connections 2021-03-09T09:35:25.056 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 1 -- 172.21.15.143:0/2398214479 >> 172.21.15.143:0/2398214479 conn(0x7f1f8002b700 msgr2=0x7f1f80006a30 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:25.056 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 1 -- 172.21.15.143:0/2398214479 shutdown_connections 2021-03-09T09:35:25.056 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 1 -- 172.21.15.143:0/2398214479 wait complete. 2021-03-09T09:35:25.057 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 10 monclient: build_initial_monmap 2021-03-09T09:35:25.057 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 10 monclient: monmap: 2021-03-09T09:35:25.057 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2021-03-09T09:35:25.057 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:25.058 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2021-03-09T09:35:25.058 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2021-03-09T09:35:25.059 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:25.059 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:25.059 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:25.060 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:25.060 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:25.060 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:25.060 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:25.061 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:25.061 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:25.061 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:25.062 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:25.062 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:25.062 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 1 Processor -- start 2021-03-09T09:35:25.063 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 1 -- start start 2021-03-09T09:35:25.064 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.006+0000 7f1f87fff700 10 monclient: init 2021-03-09T09:35:25.064 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:25.064 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient: _add_conns ranks=[4,7,1,0,6,3,2,5,8] 2021-03-09T09:35:25.065 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient(hunting): picked mon.noname-c con 0x7f1f800bb210 addr v1:172.21.15.143:6791/0 2021-03-09T09:35:25.065 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient(hunting): picked mon.noname-i con 0x7f1f800bb5f0 addr v1:172.21.15.173:6792/0 2021-03-09T09:35:25.066 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient(hunting): picked mon.noname-f con 0x7f1f800c17c0 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:25.067 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800b7060 con 0x7f1f800c17c0 2021-03-09T09:35:25.068 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 1 -- --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800bc5a0 con 0x7f1f800bb210 2021-03-09T09:35:25.069 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 1 -- --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800b8750 con 0x7f1f800bb5f0 2021-03-09T09:35:25.070 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f86ffd700 1 --1- >> v1:172.21.15.143:6791/0 conn(0x7f1f800bb210 0x7f1f800c0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6791/0 says I am v1:172.21.15.143:37462/0 (socket says 172.21.15.143:37462) 2021-03-09T09:35:25.072 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:25.072 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f86ffd700 1 -- 172.21.15.143:0/1114789348 learned_addr learned my addr 172.21.15.143:0/1114789348 (peer_addr_for_me v1:172.21.15.143:0/0) 2021-03-09T09:35:25.073 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient(hunting): authenticate will time out at 714.417317s 2021-03-09T09:35:25.074 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450614943 0 0) 0x7f1f800bc5a0 con 0x7f1f800bb210 2021-03-09T09:35:25.075 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:25.076 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): my global_id is 4153 2021-03-09T09:35:25.077 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 --> v1:172.21.15.143:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f68003230 con 0x7f1f800bb210 2021-03-09T09:35:25.078 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 541097119 0 0) 0x7f1f68003230 con 0x7f1f800bb210 2021-03-09T09:35:25.079 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 --> v1:172.21.15.143:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f800bc5a0 con 0x7f1f800bb210 2021-03-09T09:35:25.080 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f1f7c002c00 con 0x7f1f800bb210 2021-03-09T09:35:25.082 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:25.083 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T09:35:25.084 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): dump: 2021-03-09T09:35:25.085 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2021-03-09T09:35:25.087 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:25.089 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:25.091 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:25.091 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:25.091 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:25.092 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:25.092 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:25.092 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:25.092 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:25.093 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:25.093 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:25.093 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:25.093 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:25.093 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:25.094 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:25.094 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:25.094 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.7 v1:172.21.15.173:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102362979 0 0) 0x7f1f800b8750 con 0x7f1f800bb5f0 2021-03-09T09:35:25.094 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:25.095 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): my global_id is 4165 2021-03-09T09:35:25.095 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 --> v1:172.21.15.173:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f68003230 con 0x7f1f800bb5f0 2021-03-09T09:35:25.095 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022212916 0 0) 0x7f1f800b7060 con 0x7f1f800c17c0 2021-03-09T09:35:25.095 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:25.095 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient(hunting): my global_id is 4186 2021-03-09T09:35:25.096 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f800b8750 con 0x7f1f800c17c0 2021-03-09T09:35:25.096 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1164966397 0 0) 0x7f1f800bc5a0 con 0x7f1f800bb210 2021-03-09T09:35:25.096 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 >> v1:172.21.15.173:6792/0 conn(0x7f1f800bb5f0 legacy=0x7f1f800ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:25.096 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 >> v1:172.21.15.173:6789/0 conn(0x7f1f800c17c0 legacy=0x7f1f800d25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:25.096 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient: _finish_hunting 0 2021-03-09T09:35:25.097 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 monclient: found mon.c 2021-03-09T09:35:25.097 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:25.097 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:25.097 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 --> v1:172.21.15.143:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f800b7c20 con 0x7f1f800bb210 2021-03-09T09:35:25.097 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient: _finish_auth 0 2021-03-09T09:35:25.098 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:25.098 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 5 monclient: authenticate success, global_id 4153 2021-03-09T09:35:25.098 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient: _renew_subs 2021-03-09T09:35:25.098 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:25.098 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 1 -- 172.21.15.143:0/1114789348 --> v1:172.21.15.143:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f800b5510 con 0x7f1f800bb210 2021-03-09T09:35:25.099 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient: _renew_subs 2021-03-09T09:35:25.099 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:25.099 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 1 -- 172.21.15.143:0/1114789348 --> v1:172.21.15.143:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f800caf80 con 0x7f1f800bb210 2021-03-09T09:35:25.099 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f7c001730 con 0x7f1f800bb210 2021-03-09T09:35:25.100 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:25.100 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f1f7c001c60 con 0x7f1f800bb210 2021-03-09T09:35:25.100 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f857fa700 4 set_mon_vals no callback set 2021-03-09T09:35:25.100 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:25.100 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:25.101 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 10 monclient: dump: 2021-03-09T09:35:25.101 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2021-03-09T09:35:25.101 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:25.101 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:25.101 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:25.102 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:25.102 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:25.102 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:25.102 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:25.102 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:25.103 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:25.103 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:25.103 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:25.103 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:25.103 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:25.104 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:25.104 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:25.104 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 7 ==== mgrmap(e 1) v1 ==== 623+0+0 (unknown 1065898531 0 0) 0x7f1f7c002120 con 0x7f1f800bb210 2021-03-09T09:35:25.104 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (unknown 3931405141 0 0) 0x7f1f7c0060e0 con 0x7f1f800bb210 2021-03-09T09:35:25.104 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:25.105 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:25.105 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:25.105 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.010+0000 7f1f87fff700 1 -- 172.21.15.143:0/1114789348 --> v1:172.21.15.143:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f60004d50 con 0x7f1f800bb210 2021-03-09T09:35:25.105 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.014+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (unknown 1092875540 0 637523725) 0x7f1f7c006390 con 0x7f1f800bb210 2021-03-09T09:35:25.106 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.014+0000 7f1f72ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:25.106 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.014+0000 7f1f72ffd700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:25.107 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.082+0000 7f1f87fff700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-03-09T09:35:25.107 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.082+0000 7f1f87fff700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-03-09T09:35:25.107 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.082+0000 7f1f87fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:25.107 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.082+0000 7f1f87fff700 1 -- 172.21.15.143:0/1114789348 --> v1:172.21.15.143:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f60005000 con 0x7f1f800bb210 2021-03-09T09:35:25.108 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.082+0000 7f1f72ffd700 1 -- 172.21.15.143:0/1114789348 <== mon.4 v1:172.21.15.143:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5570 (unknown 150273500 0 372050164) 0x7f1f7c0056c0 con 0x7f1f800bb210 2021-03-09T09:35:25.108 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.082+0000 7f1f72ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-03-09T09:35:25.108 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.082+0000 7f1f72ffd700 10 monclient: _finish_command 2 = system:0 2021-03-09T09:35:25.108 INFO:teuthology.misc.health.smithi143.stdout: 2021-03-09T09:35:25.109 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":8,"fsid":"8a34d6d4-5d27-4c75-9119-8e930fb2bed7","created":"2021-03-09T09:35:21.903074+0000","modified":"2021-03-09T09:35:24.182490+0000","last_up_change":"0.000000","last_in_change":"2021-03-09T09:35:24.182490+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"72ccc8f8-8bb0-4230-bfea-c47a8dadaf1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dd598087-2308-4d24-9bbc-a6213e89ca6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a8112181-081d-43e4-af54-f8388903c61a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"23b46e8e-3d64-48b2-be36-1594df839111","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e50766be-3a46-4493-bb02-6eba34d6437d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"1169fffe-55ed-4cf1-b718-b9bf3d9ffe2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-09T09:35:25.110 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.086+0000 7f1f87fff700 10 monclient: shutdown 2021-03-09T09:35:25.110 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.086+0000 7f1f87fff700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:25.111 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.086+0000 7f1f87fff700 1 -- 172.21.15.143:0/1114789348 >> v1:172.21.15.143:6791/0 conn(0x7f1f800bb210 legacy=0x7f1f800c0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:25.111 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.086+0000 7f1f87fff700 1 -- 172.21.15.143:0/1114789348 shutdown_connections 2021-03-09T09:35:25.111 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.086+0000 7f1f87fff700 1 -- 172.21.15.143:0/1114789348 >> 172.21.15.143:0/1114789348 conn(0x7f1f8002b700 msgr2=0x7f1f800b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:25.111 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.086+0000 7f1f87fff700 1 -- 172.21.15.143:0/1114789348 shutdown_connections 2021-03-09T09:35:25.112 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:25.086+0000 7f1f87fff700 1 -- 172.21.15.143:0/1114789348 wait complete. 2021-03-09T09:35:25.112 DEBUG:teuthology.misc:0 of 6 OSDs are up 2021-03-09T09:35:28.442 INFO:tasks.ceph.osd.0.smithi143.stderr:2021-03-09T09:35:28.442+0000 7f28c0f50f00 -1 osd.0 0 log_to_monitors {default=true} 2021-03-09T09:35:28.575 INFO:tasks.ceph.osd.3.smithi173.stderr:2021-03-09T09:35:28.573+0000 7f02a938ff00 -1 osd.3 0 log_to_monitors {default=true} 2021-03-09T09:35:28.660 INFO:tasks.ceph.osd.1.smithi143.stderr:2021-03-09T09:35:28.658+0000 7f471cef6f00 -1 osd.1 0 log_to_monitors {default=true} 2021-03-09T09:35:28.673 INFO:tasks.ceph.osd.5.smithi173.stderr:2021-03-09T09:35:28.673+0000 7f3b102f0f00 -1 osd.5 0 log_to_monitors {default=true} 2021-03-09T09:35:28.693 INFO:tasks.ceph.osd.4.smithi173.stderr:2021-03-09T09:35:28.693+0000 7fa740c4df00 -1 osd.4 0 log_to_monitors {default=true} 2021-03-09T09:35:28.739 INFO:tasks.ceph.osd.2.smithi143.stderr:2021-03-09T09:35:28.738+0000 7ff44474df00 -1 osd.2 0 log_to_monitors {default=true} 2021-03-09T09:35:30.107 INFO:tasks.ceph.osd.3.smithi173.stderr:2021-03-09T09:35:30.105+0000 7f02a0859700 -1 osd.3 0 waiting for initial osdmap 2021-03-09T09:35:30.107 INFO:tasks.ceph.osd.0.smithi143.stderr:2021-03-09T09:35:30.106+0000 7f28b841a700 -1 osd.0 0 waiting for initial osdmap 2021-03-09T09:35:30.114 INFO:tasks.ceph.osd.5.smithi173.stderr:2021-03-09T09:35:30.113+0000 7f3b077ba700 -1 osd.5 0 waiting for initial osdmap 2021-03-09T09:35:30.116 INFO:tasks.ceph.osd.2.smithi143.stderr:2021-03-09T09:35:30.114+0000 7ff43d41a700 -1 osd.2 0 waiting for initial osdmap 2021-03-09T09:35:30.117 INFO:tasks.ceph.osd.1.smithi143.stderr:2021-03-09T09:35:30.114+0000 7f47143c0700 -1 osd.1 0 waiting for initial osdmap 2021-03-09T09:35:30.117 INFO:tasks.ceph.osd.4.smithi173.stderr:2021-03-09T09:35:30.117+0000 7fa738117700 -1 osd.4 0 waiting for initial osdmap 2021-03-09T09:35:31.189 INFO:tasks.ceph.mon.a.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f6446afc700 time 2021-03-09T09:35:31.192338+0000 2021-03-09T09:35:31.189 INFO:tasks.ceph.mon.a.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:35:31.199 INFO:tasks.ceph.mon.a.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:31.199 INFO:tasks.ceph.mon.a.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f64532be3dd] 2021-03-09T09:35:31.199 INFO:tasks.ceph.mon.a.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f64532be5b8] 2021-03-09T09:35:31.200 INFO:tasks.ceph.mon.a.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b14c8dda65] 2021-03-09T09:35:31.200 INFO:tasks.ceph.mon.a.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b14c929a71] 2021-03-09T09:35:31.200 INFO:tasks.ceph.mon.a.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b14c959295] 2021-03-09T09:35:31.200 INFO:tasks.ceph.mon.a.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b14c96124b] 2021-03-09T09:35:31.201 INFO:tasks.ceph.mon.a.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b14c9617ab] 2021-03-09T09:35:31.201 INFO:tasks.ceph.mon.a.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b14c8d0ade] 2021-03-09T09:35:31.201 INFO:tasks.ceph.mon.a.smithi143.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55b14c7bbc79] 2021-03-09T09:35:31.201 INFO:tasks.ceph.mon.a.smithi143.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55b14c7bf229] 2021-03-09T09:35:31.201 INFO:tasks.ceph.mon.a.smithi143.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.202 INFO:tasks.ceph.mon.a.smithi143.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55b14c7c187f] 2021-03-09T09:35:31.202 INFO:tasks.ceph.mon.a.smithi143.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55b14c7bfd70] 2021-03-09T09:35:31.202 INFO:tasks.ceph.mon.a.smithi143.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.202 INFO:tasks.ceph.mon.a.smithi143.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55b14c7ee178] 2021-03-09T09:35:31.203 INFO:tasks.ceph.mon.a.smithi143.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7f64534c02ca] 2021-03-09T09:35:31.203 INFO:tasks.ceph.mon.a.smithi143.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f645356a71d] 2021-03-09T09:35:31.203 INFO:tasks.ceph.mon.a.smithi143.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f645212e6db] 2021-03-09T09:35:31.203 INFO:tasks.ceph.mon.a.smithi143.stderr: 19: clone() 2021-03-09T09:35:31.204 INFO:tasks.ceph.mon.a.smithi143.stderr:*** Caught signal (Aborted) ** 2021-03-09T09:35:31.204 INFO:tasks.ceph.mon.a.smithi143.stderr: in thread 7f6446afc700 thread_name:ms_dispatch 2021-03-09T09:35:31.204 INFO:tasks.ceph.mon.a.smithi143.stderr:2021-03-09T09:35:31.198+0000 7f6446afc700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f6446afc700 time 2021-03-09T09:35:31.192338+0000 2021-03-09T09:35:31.204 INFO:tasks.ceph.mon.a.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:35:31.204 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.205 INFO:tasks.ceph.mon.a.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:31.205 INFO:tasks.ceph.mon.a.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f64532be3dd] 2021-03-09T09:35:31.205 INFO:tasks.ceph.mon.a.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f64532be5b8] 2021-03-09T09:35:31.205 INFO:tasks.ceph.mon.a.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b14c8dda65] 2021-03-09T09:35:31.206 INFO:tasks.ceph.mon.a.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b14c929a71] 2021-03-09T09:35:31.206 INFO:tasks.ceph.mon.a.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b14c959295] 2021-03-09T09:35:31.206 INFO:tasks.ceph.mon.a.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b14c96124b] 2021-03-09T09:35:31.206 INFO:tasks.ceph.mon.a.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b14c9617ab] 2021-03-09T09:35:31.207 INFO:tasks.ceph.mon.a.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b14c8d0ade] 2021-03-09T09:35:31.207 INFO:tasks.ceph.mon.a.smithi143.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55b14c7bbc79] 2021-03-09T09:35:31.207 INFO:tasks.ceph.mon.a.smithi143.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55b14c7bf229] 2021-03-09T09:35:31.208 INFO:tasks.ceph.mon.a.smithi143.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.208 INFO:tasks.ceph.mon.a.smithi143.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55b14c7c187f] 2021-03-09T09:35:31.208 INFO:tasks.ceph.mon.a.smithi143.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55b14c7bfd70] 2021-03-09T09:35:31.208 INFO:tasks.ceph.mon.a.smithi143.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.208 INFO:tasks.ceph.mon.a.smithi143.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55b14c7ee178] 2021-03-09T09:35:31.209 INFO:tasks.ceph.mon.a.smithi143.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7f64534c02ca] 2021-03-09T09:35:31.209 INFO:tasks.ceph.mon.a.smithi143.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f645356a71d] 2021-03-09T09:35:31.209 INFO:tasks.ceph.mon.a.smithi143.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f645212e6db] 2021-03-09T09:35:31.209 INFO:tasks.ceph.mon.a.smithi143.stderr: 19: clone() 2021-03-09T09:35:31.210 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.210 INFO:tasks.ceph.mon.a.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:31.211 INFO:tasks.ceph.mon.a.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6452139980] 2021-03-09T09:35:31.211 INFO:tasks.ceph.mon.a.smithi143.stderr: 2: gsignal() 2021-03-09T09:35:31.211 INFO:tasks.ceph.mon.a.smithi143.stderr: 3: abort() 2021-03-09T09:35:31.211 INFO:tasks.ceph.mon.a.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f64532be42e] 2021-03-09T09:35:31.212 INFO:tasks.ceph.mon.a.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f64532be5b8] 2021-03-09T09:35:31.212 INFO:tasks.ceph.mon.a.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b14c8dda65] 2021-03-09T09:35:31.212 INFO:tasks.ceph.mon.a.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b14c929a71] 2021-03-09T09:35:31.212 INFO:tasks.ceph.mon.a.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b14c959295] 2021-03-09T09:35:31.213 INFO:tasks.ceph.mon.a.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b14c96124b] 2021-03-09T09:35:31.213 INFO:tasks.ceph.mon.a.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b14c9617ab] 2021-03-09T09:35:31.213 INFO:tasks.ceph.mon.a.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b14c8d0ade] 2021-03-09T09:35:31.213 INFO:tasks.ceph.mon.a.smithi143.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55b14c7bbc79] 2021-03-09T09:35:31.214 INFO:tasks.ceph.mon.a.smithi143.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55b14c7bf229] 2021-03-09T09:35:31.214 INFO:tasks.ceph.mon.a.smithi143.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.214 INFO:tasks.ceph.mon.a.smithi143.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55b14c7c187f] 2021-03-09T09:35:31.214 INFO:tasks.ceph.mon.a.smithi143.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55b14c7bfd70] 2021-03-09T09:35:31.215 INFO:tasks.ceph.mon.a.smithi143.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.215 INFO:tasks.ceph.mon.a.smithi143.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55b14c7ee178] 2021-03-09T09:35:31.215 INFO:tasks.ceph.mon.a.smithi143.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f64534c02ca] 2021-03-09T09:35:31.215 INFO:tasks.ceph.mon.a.smithi143.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f645356a71d] 2021-03-09T09:35:31.216 INFO:tasks.ceph.mon.a.smithi143.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f645212e6db] 2021-03-09T09:35:31.216 INFO:tasks.ceph.mon.a.smithi143.stderr: 22: clone() 2021-03-09T09:35:31.216 INFO:tasks.ceph.mon.a.smithi143.stderr:2021-03-09T09:35:31.206+0000 7f6446afc700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:35:31.216 INFO:tasks.ceph.mon.a.smithi143.stderr: in thread 7f6446afc700 thread_name:ms_dispatch 2021-03-09T09:35:31.217 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.217 INFO:tasks.ceph.mon.a.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:31.217 INFO:tasks.ceph.mon.a.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6452139980] 2021-03-09T09:35:31.217 INFO:tasks.ceph.mon.a.smithi143.stderr: 2: gsignal() 2021-03-09T09:35:31.218 INFO:tasks.ceph.mon.a.smithi143.stderr: 3: abort() 2021-03-09T09:35:31.218 INFO:tasks.ceph.mon.a.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f64532be42e] 2021-03-09T09:35:31.218 INFO:tasks.ceph.mon.a.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f64532be5b8] 2021-03-09T09:35:31.218 INFO:tasks.ceph.mon.a.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b14c8dda65] 2021-03-09T09:35:31.219 INFO:tasks.ceph.mon.a.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b14c929a71] 2021-03-09T09:35:31.219 INFO:tasks.ceph.mon.a.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b14c959295] 2021-03-09T09:35:31.219 INFO:tasks.ceph.mon.a.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b14c96124b] 2021-03-09T09:35:31.219 INFO:tasks.ceph.mon.a.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b14c9617ab] 2021-03-09T09:35:31.220 INFO:tasks.ceph.mon.a.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b14c8d0ade] 2021-03-09T09:35:31.220 INFO:tasks.ceph.mon.a.smithi143.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55b14c7bbc79] 2021-03-09T09:35:31.220 INFO:tasks.ceph.mon.a.smithi143.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55b14c7bf229] 2021-03-09T09:35:31.220 INFO:tasks.ceph.mon.a.smithi143.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.221 INFO:tasks.ceph.mon.a.smithi143.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55b14c7c187f] 2021-03-09T09:35:31.221 INFO:tasks.ceph.mon.a.smithi143.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55b14c7bfd70] 2021-03-09T09:35:31.221 INFO:tasks.ceph.mon.a.smithi143.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.221 INFO:tasks.ceph.mon.a.smithi143.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55b14c7ee178] 2021-03-09T09:35:31.221 INFO:tasks.ceph.mon.a.smithi143.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f64534c02ca] 2021-03-09T09:35:31.222 INFO:tasks.ceph.mon.a.smithi143.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f645356a71d] 2021-03-09T09:35:31.222 INFO:tasks.ceph.mon.a.smithi143.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f645212e6db] 2021-03-09T09:35:31.222 INFO:tasks.ceph.mon.a.smithi143.stderr: 22: clone() 2021-03-09T09:35:31.222 INFO:tasks.ceph.mon.a.smithi143.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:35:31.223 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.223 INFO:tasks.ceph.mon.a.smithi143.stderr: -1> 2021-03-09T09:35:31.198+0000 7f6446afc700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f6446afc700 time 2021-03-09T09:35:31.192338+0000 2021-03-09T09:35:31.224 INFO:tasks.ceph.mon.a.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:35:31.224 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.224 INFO:tasks.ceph.mon.a.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:31.224 INFO:tasks.ceph.mon.a.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f64532be3dd] 2021-03-09T09:35:31.225 INFO:tasks.ceph.mon.a.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f64532be5b8] 2021-03-09T09:35:31.225 INFO:tasks.ceph.mon.a.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b14c8dda65] 2021-03-09T09:35:31.225 INFO:tasks.ceph.mon.a.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b14c929a71] 2021-03-09T09:35:31.225 INFO:tasks.ceph.mon.a.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b14c959295] 2021-03-09T09:35:31.226 INFO:tasks.ceph.mon.a.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b14c96124b] 2021-03-09T09:35:31.226 INFO:tasks.ceph.mon.a.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b14c9617ab] 2021-03-09T09:35:31.226 INFO:tasks.ceph.mon.a.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b14c8d0ade] 2021-03-09T09:35:31.226 INFO:tasks.ceph.mon.a.smithi143.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55b14c7bbc79] 2021-03-09T09:35:31.227 INFO:tasks.ceph.mon.a.smithi143.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55b14c7bf229] 2021-03-09T09:35:31.227 INFO:tasks.ceph.mon.a.smithi143.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.227 INFO:tasks.ceph.mon.a.smithi143.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55b14c7c187f] 2021-03-09T09:35:31.228 INFO:tasks.ceph.mon.a.smithi143.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55b14c7bfd70] 2021-03-09T09:35:31.228 INFO:tasks.ceph.mon.a.smithi143.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.228 INFO:tasks.ceph.mon.a.smithi143.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55b14c7ee178] 2021-03-09T09:35:31.228 INFO:tasks.ceph.mon.a.smithi143.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7f64534c02ca] 2021-03-09T09:35:31.229 INFO:tasks.ceph.mon.a.smithi143.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f645356a71d] 2021-03-09T09:35:31.229 INFO:tasks.ceph.mon.a.smithi143.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f645212e6db] 2021-03-09T09:35:31.229 INFO:tasks.ceph.mon.a.smithi143.stderr: 19: clone() 2021-03-09T09:35:31.230 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.230 INFO:tasks.ceph.mon.a.smithi143.stderr: 0> 2021-03-09T09:35:31.206+0000 7f6446afc700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:35:31.230 INFO:tasks.ceph.mon.a.smithi143.stderr: in thread 7f6446afc700 thread_name:ms_dispatch 2021-03-09T09:35:31.230 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.230 INFO:tasks.ceph.mon.a.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:31.231 INFO:tasks.ceph.mon.a.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6452139980] 2021-03-09T09:35:31.231 INFO:tasks.ceph.mon.a.smithi143.stderr: 2: gsignal() 2021-03-09T09:35:31.231 INFO:tasks.ceph.mon.a.smithi143.stderr: 3: abort() 2021-03-09T09:35:31.231 INFO:tasks.ceph.mon.a.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f64532be42e] 2021-03-09T09:35:31.232 INFO:tasks.ceph.mon.a.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f64532be5b8] 2021-03-09T09:35:31.232 INFO:tasks.ceph.mon.a.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b14c8dda65] 2021-03-09T09:35:31.232 INFO:tasks.ceph.mon.a.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b14c929a71] 2021-03-09T09:35:31.232 INFO:tasks.ceph.mon.a.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b14c959295] 2021-03-09T09:35:31.233 INFO:tasks.ceph.mon.a.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b14c96124b] 2021-03-09T09:35:31.233 INFO:tasks.ceph.mon.a.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b14c9617ab] 2021-03-09T09:35:31.233 INFO:tasks.ceph.mon.a.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b14c8d0ade] 2021-03-09T09:35:31.233 INFO:tasks.ceph.mon.a.smithi143.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55b14c7bbc79] 2021-03-09T09:35:31.233 INFO:tasks.ceph.mon.a.smithi143.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55b14c7bf229] 2021-03-09T09:35:31.234 INFO:tasks.ceph.mon.a.smithi143.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.234 INFO:tasks.ceph.mon.a.smithi143.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55b14c7c187f] 2021-03-09T09:35:31.234 INFO:tasks.ceph.mon.a.smithi143.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55b14c7bfd70] 2021-03-09T09:35:31.234 INFO:tasks.ceph.mon.a.smithi143.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.235 INFO:tasks.ceph.mon.a.smithi143.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55b14c7ee178] 2021-03-09T09:35:31.235 INFO:tasks.ceph.mon.a.smithi143.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f64534c02ca] 2021-03-09T09:35:31.235 INFO:tasks.ceph.mon.a.smithi143.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f645356a71d] 2021-03-09T09:35:31.235 INFO:tasks.ceph.mon.a.smithi143.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f645212e6db] 2021-03-09T09:35:31.236 INFO:tasks.ceph.mon.a.smithi143.stderr: 22: clone() 2021-03-09T09:35:31.236 INFO:tasks.ceph.mon.a.smithi143.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:35:31.236 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.237 INFO:tasks.ceph.mon.a.smithi143.stderr: -1> 2021-03-09T09:35:31.198+0000 7f6446afc700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f6446afc700 time 2021-03-09T09:35:31.192338+0000 2021-03-09T09:35:31.237 INFO:tasks.ceph.mon.a.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:35:31.237 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.238 INFO:tasks.ceph.mon.a.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:31.238 INFO:tasks.ceph.mon.a.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f64532be3dd] 2021-03-09T09:35:31.238 INFO:tasks.ceph.mon.a.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f64532be5b8] 2021-03-09T09:35:31.238 INFO:tasks.ceph.mon.a.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b14c8dda65] 2021-03-09T09:35:31.239 INFO:tasks.ceph.mon.a.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b14c929a71] 2021-03-09T09:35:31.239 INFO:tasks.ceph.mon.a.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b14c959295] 2021-03-09T09:35:31.239 INFO:tasks.ceph.mon.a.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b14c96124b] 2021-03-09T09:35:31.239 INFO:tasks.ceph.mon.a.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b14c9617ab] 2021-03-09T09:35:31.240 INFO:tasks.ceph.mon.a.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b14c8d0ade] 2021-03-09T09:35:31.240 INFO:tasks.ceph.mon.a.smithi143.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55b14c7bbc79] 2021-03-09T09:35:31.240 INFO:tasks.ceph.mon.a.smithi143.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55b14c7bf229] 2021-03-09T09:35:31.240 INFO:tasks.ceph.mon.a.smithi143.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.241 INFO:tasks.ceph.mon.a.smithi143.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55b14c7c187f] 2021-03-09T09:35:31.241 INFO:tasks.ceph.mon.a.smithi143.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55b14c7bfd70] 2021-03-09T09:35:31.241 INFO:tasks.ceph.mon.a.smithi143.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.241 INFO:tasks.ceph.mon.a.smithi143.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55b14c7ee178] 2021-03-09T09:35:31.241 INFO:tasks.ceph.mon.a.smithi143.stderr: 16: (DispatchQueue::entry()+0x11ea) [0x7f64534c02ca] 2021-03-09T09:35:31.242 INFO:tasks.ceph.mon.a.smithi143.stderr: 17: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f645356a71d] 2021-03-09T09:35:31.242 INFO:tasks.ceph.mon.a.smithi143.stderr: 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f645212e6db] 2021-03-09T09:35:31.242 INFO:tasks.ceph.mon.a.smithi143.stderr: 19: clone() 2021-03-09T09:35:31.242 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.243 INFO:tasks.ceph.mon.a.smithi143.stderr: 0> 2021-03-09T09:35:31.206+0000 7f6446afc700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:35:31.243 INFO:tasks.ceph.mon.a.smithi143.stderr: in thread 7f6446afc700 thread_name:ms_dispatch 2021-03-09T09:35:31.243 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.243 INFO:tasks.ceph.mon.a.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:31.243 INFO:tasks.ceph.mon.a.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6452139980] 2021-03-09T09:35:31.244 INFO:tasks.ceph.mon.a.smithi143.stderr: 2: gsignal() 2021-03-09T09:35:31.244 INFO:tasks.ceph.mon.a.smithi143.stderr: 3: abort() 2021-03-09T09:35:31.244 INFO:tasks.ceph.mon.a.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f64532be42e] 2021-03-09T09:35:31.244 INFO:tasks.ceph.mon.a.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f64532be5b8] 2021-03-09T09:35:31.245 INFO:tasks.ceph.mon.a.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b14c8dda65] 2021-03-09T09:35:31.245 INFO:tasks.ceph.mon.a.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b14c929a71] 2021-03-09T09:35:31.245 INFO:tasks.ceph.mon.a.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b14c959295] 2021-03-09T09:35:31.246 INFO:tasks.ceph.mon.a.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b14c96124b] 2021-03-09T09:35:31.246 INFO:tasks.ceph.mon.a.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b14c9617ab] 2021-03-09T09:35:31.246 INFO:tasks.ceph.mon.a.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b14c8d0ade] 2021-03-09T09:35:31.246 INFO:tasks.ceph.mon.a.smithi143.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x47d9) [0x55b14c7bbc79] 2021-03-09T09:35:31.247 INFO:tasks.ceph.mon.a.smithi143.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x849) [0x55b14c7bf229] 2021-03-09T09:35:31.247 INFO:tasks.ceph.mon.a.smithi143.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.247 INFO:tasks.ceph.mon.a.smithi143.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x68f) [0x55b14c7c187f] 2021-03-09T09:35:31.248 INFO:tasks.ceph.mon.a.smithi143.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1390) [0x55b14c7bfd70] 2021-03-09T09:35:31.248 INFO:tasks.ceph.mon.a.smithi143.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x55b14c7c066a] 2021-03-09T09:35:31.248 INFO:tasks.ceph.mon.a.smithi143.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x55b14c7ee178] 2021-03-09T09:35:31.249 INFO:tasks.ceph.mon.a.smithi143.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f64534c02ca] 2021-03-09T09:35:31.249 INFO:tasks.ceph.mon.a.smithi143.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f645356a71d] 2021-03-09T09:35:31.249 INFO:tasks.ceph.mon.a.smithi143.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f645212e6db] 2021-03-09T09:35:31.249 INFO:tasks.ceph.mon.a.smithi143.stderr: 22: clone() 2021-03-09T09:35:31.250 INFO:tasks.ceph.mon.a.smithi143.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:35:31.250 INFO:tasks.ceph.mon.a.smithi143.stderr: 2021-03-09T09:35:31.727 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T09:35:31.905 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient: get_monmap_and_config 2021-03-09T09:35:31.906 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient: build_initial_monmap 2021-03-09T09:35:31.906 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient: monmap: 2021-03-09T09:35:31.907 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2021-03-09T09:35:31.907 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:31.908 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:35:31.908842+0000 2021-03-09T09:35:31.909 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:35:31.908842+0000 2021-03-09T09:35:31.909 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:31.910 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:31.910 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:31.910 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:31.911 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:31.911 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:31.912 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:31.912 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:31.912 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:31.913 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:31.914 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:31.915 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:31.929 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 Processor -- start 2021-03-09T09:35:31.929 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 -- start start 2021-03-09T09:35:31.929 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient: init 2021-03-09T09:35:31.930 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:31.930 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient: _add_conns ranks=[5,6,2,3,7,8,0,1,4] 2021-03-09T09:35:31.930 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient(hunting): picked mon.noname-h con 0x7f3d5409b6e0 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:31.931 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient(hunting): picked mon.noname-d con 0x7f3d540999c0 addr v1:172.21.15.143:6792/0 2021-03-09T09:35:31.931 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient(hunting): picked mon.noname-b con 0x7f3d54099e70 addr v1:172.21.15.143:6790/0 2021-03-09T09:35:31.931 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d540460b0 con 0x7f3d54099e70 2021-03-09T09:35:31.932 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5409b1f0 con 0x7f3d5409b6e0 2021-03-09T09:35:31.932 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 -- --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d54090300 con 0x7f3d540999c0 2021-03-09T09:35:31.932 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:31.932 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient(hunting): authenticate will time out at 721.313772s 2021-03-09T09:35:31.933 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d537fe700 1 --1- >> v1:172.21.15.143:6792/0 conn(0x7f3d540999c0 0x7f3d54130790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6792/0 says I am v1:172.21.15.143:52634/0 (socket says 172.21.15.143:52634) 2021-03-09T09:35:31.933 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d537fe700 1 -- 172.21.15.143:0/2306720538 learned_addr learned my addr 172.21.15.143:0/2306720538 (peer_addr_for_me v1:172.21.15.143:0/0) 2021-03-09T09:35:31.933 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 <== mon.6 v1:172.21.15.143:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054642785 0 0) 0x7f3d54090300 con 0x7f3d540999c0 2021-03-09T09:35:31.933 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:31.934 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient(hunting): my global_id is 4218 2021-03-09T09:35:31.934 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 --> v1:172.21.15.143:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d40003230 con 0x7f3d540999c0 2021-03-09T09:35:31.934 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970400 0 0) 0x7f3d540460b0 con 0x7f3d54099e70 2021-03-09T09:35:31.934 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:31.934 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient(hunting): my global_id is 4187 2021-03-09T09:35:31.935 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d54090300 con 0x7f3d54099e70 2021-03-09T09:35:31.935 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529562067 0 0) 0x7f3d5409b1f0 con 0x7f3d5409b6e0 2021-03-09T09:35:31.935 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:31.935 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient(hunting): my global_id is 4271 2021-03-09T09:35:31.936 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d540460b0 con 0x7f3d5409b6e0 2021-03-09T09:35:31.936 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 <== mon.6 v1:172.21.15.143:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2546981732 0 0) 0x7f3d40003230 con 0x7f3d540999c0 2021-03-09T09:35:31.936 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 >> v1:172.21.15.173:6791/0 conn(0x7f3d5409b6e0 legacy=0x7f3d540992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:31.936 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 >> v1:172.21.15.143:6790/0 conn(0x7f3d54099e70 legacy=0x7f3d54133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:31.937 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: _finish_hunting 0 2021-03-09T09:35:31.937 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 monclient: found mon.noname-d 2021-03-09T09:35:31.937 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:31.938 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.143:6792/0 2021-03-09T09:35:31.938 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 --> v1:172.21.15.143:6792/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d540905a0 con 0x7f3d540999c0 2021-03-09T09:35:31.938 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: _finish_auth 0 2021-03-09T09:35:31.938 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:31.939 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 <== mon.6 v1:172.21.15.143:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f3d3c001860 con 0x7f3d540999c0 2021-03-09T09:35:31.939 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:31.939 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2021-03-09T09:35:31.939 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: dump: 2021-03-09T09:35:31.943 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2021-03-09T09:35:31.944 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:31.944 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:31.944 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:31.945 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:31.945 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:31.945 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:31.945 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:31.945 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:31.946 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:31.946 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:31.946 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:31.946 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:31.946 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:31.947 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:31.947 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:31.947 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 5 monclient: authenticate success, global_id 4218 2021-03-09T09:35:31.947 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:31.948 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 <== mon.6 v1:172.21.15.143:6792/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d3c001a80 con 0x7f3d540999c0 2021-03-09T09:35:31.948 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:31.948 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:31.948 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 4 set_mon_vals no callback set 2021-03-09T09:35:31.948 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 1 -- 172.21.15.143:0/2306720538 <== mon.6 v1:172.21.15.143:6792/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f3d3c0048f0 con 0x7f3d540999c0 2021-03-09T09:35:31.949 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:31.949 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: got monmap 1 from mon.d (according to old e1) 2021-03-09T09:35:31.949 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d52ffd700 10 monclient: dump: 2021-03-09T09:35:31.949 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2021-03-09T09:35:31.950 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:31.950 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:31.950 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:31.950 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:31.950 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:31.951 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:31.951 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:31.951 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:31.951 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:31.951 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:31.952 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:31.952 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:31.952 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:31.952 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:31.953 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:31.953 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 10 monclient: shutdown 2021-03-09T09:35:31.953 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:31.953 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 -- 172.21.15.143:0/2306720538 >> v1:172.21.15.143:6792/0 conn(0x7f3d540999c0 legacy=0x7f3d54130790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:31.954 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 -- 172.21.15.143:0/2306720538 shutdown_connections 2021-03-09T09:35:31.954 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 -- 172.21.15.143:0/2306720538 >> 172.21.15.143:0/2306720538 conn(0x7f3d5406a240 msgr2=0x7f3d5406a650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:31.954 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 -- 172.21.15.143:0/2306720538 shutdown_connections 2021-03-09T09:35:31.954 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.906+0000 7f3d592be700 1 -- 172.21.15.143:0/2306720538 wait complete. 2021-03-09T09:35:31.955 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient: build_initial_monmap 2021-03-09T09:35:31.955 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient: monmap: 2021-03-09T09:35:31.955 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2021-03-09T09:35:31.955 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:31.956 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2021-03-09T09:35:31.956 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2021-03-09T09:35:31.956 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:31.956 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:31.957 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:31.957 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:31.957 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:31.958 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:31.958 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:31.958 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:31.958 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:31.959 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:31.959 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:31.959 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:31.959 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 1 Processor -- start 2021-03-09T09:35:31.959 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 1 -- start start 2021-03-09T09:35:31.960 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient: init 2021-03-09T09:35:31.960 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:31.960 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient: _add_conns ranks=[0,4,5,3,6,1,8,2,7] 2021-03-09T09:35:31.960 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient(hunting): picked mon.noname-a con 0x7f3d540999c0 addr v1:172.21.15.143:6789/0 2021-03-09T09:35:31.960 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient(hunting): picked mon.noname-c con 0x7f3d54099e70 addr v1:172.21.15.143:6791/0 2021-03-09T09:35:31.961 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient(hunting): picked mon.noname-h con 0x7f3d5409b6e0 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:31.961 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d53fff700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7f3d540999c0 legacy=0x7f3d54128130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:35:31.961 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d54118660 con 0x7f3d540999c0 2021-03-09T09:35:31.961 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 1 -- --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d54118900 con 0x7f3d54099e70 2021-03-09T09:35:31.961 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d54118ba0 con 0x7f3d5409b6e0 2021-03-09T09:35:31.962 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:31.962 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient(hunting): authenticate will time out at 721.317778s 2021-03-09T09:35:31.962 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d537fe700 1 --1- >> v1:172.21.15.143:6791/0 conn(0x7f3d54099e70 0x7f3d5413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6791/0 says I am v1:172.21.15.143:37598/0 (socket says 172.21.15.143:37598) 2021-03-09T09:35:31.962 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d537fe700 1 -- 172.21.15.143:0/3904511274 learned_addr learned my addr 172.21.15.143:0/3904511274 (peer_addr_for_me v1:172.21.15.143:0/0) 2021-03-09T09:35:31.963 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254523046 0 0) 0x7f3d54118900 con 0x7f3d54099e70 2021-03-09T09:35:31.963 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:31.963 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient(hunting): my global_id is 4162 2021-03-09T09:35:31.963 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 --> v1:172.21.15.143:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34003230 con 0x7f3d54099e70 2021-03-09T09:35:31.964 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 367449518 0 0) 0x7f3d34003230 con 0x7f3d54099e70 2021-03-09T09:35:31.964 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 --> v1:172.21.15.143:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d54118900 con 0x7f3d54099e70 2021-03-09T09:35:31.964 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f3d3c004830 con 0x7f3d54099e70 2021-03-09T09:35:31.964 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:31.965 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T09:35:31.965 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient(hunting): dump: 2021-03-09T09:35:31.965 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2021-03-09T09:35:31.966 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:31.966 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:31.966 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:31.966 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:31.967 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:31.967 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:31.967 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:31.967 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:31.967 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:31.968 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:31.968 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:31.968 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:31.968 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:31.969 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:31.969 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:31.969 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:31.970 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938885703 0 0) 0x7f3d54118ba0 con 0x7f3d5409b6e0 2021-03-09T09:35:31.970 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:31.971 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient(hunting): my global_id is 4280 2021-03-09T09:35:31.971 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34003230 con 0x7f3d5409b6e0 2021-03-09T09:35:31.971 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 847948158 0 0) 0x7f3d54118900 con 0x7f3d54099e70 2021-03-09T09:35:31.971 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 >> v1:172.21.15.173:6791/0 conn(0x7f3d5409b6e0 legacy=0x7f3d54142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:31.972 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 >> v1:172.21.15.143:6789/0 conn(0x7f3d540999c0 legacy=0x7f3d54128130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:31.972 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient: _finish_hunting 0 2021-03-09T09:35:31.972 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 monclient: found mon.c 2021-03-09T09:35:31.972 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:31.973 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:31.973 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 --> v1:172.21.15.143:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5413b390 con 0x7f3d54099e70 2021-03-09T09:35:31.973 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient: _finish_auth 0 2021-03-09T09:35:31.973 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:31.973 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 5 monclient: authenticate success, global_id 4162 2021-03-09T09:35:31.974 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient: _renew_subs 2021-03-09T09:35:31.974 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:31.974 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 --> v1:172.21.15.143:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d54116a30 con 0x7f3d54099e70 2021-03-09T09:35:31.974 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient: _renew_subs 2021-03-09T09:35:31.975 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:31.975 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 --> v1:172.21.15.143:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5412cb90 con 0x7f3d54099e70 2021-03-09T09:35:31.975 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d3c001cb0 con 0x7f3d54099e70 2021-03-09T09:35:31.975 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:31.975 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7f3d3c001190 con 0x7f3d54099e70 2021-03-09T09:35:31.976 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:31.976 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d527fc700 4 set_mon_vals no callback set 2021-03-09T09:35:31.976 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:31.977 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 10 monclient: dump: 2021-03-09T09:35:31.977 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2021-03-09T09:35:31.977 INFO:teuthology.misc.health.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:31.978 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:31.978 INFO:teuthology.misc.health.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:31.978 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:31.978 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:31.979 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:31.979 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:31.979 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:31.980 INFO:teuthology.misc.health.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:31.980 INFO:teuthology.misc.health.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:31.981 INFO:teuthology.misc.health.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:31.981 INFO:teuthology.misc.health.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:31.981 INFO:teuthology.misc.health.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:31.982 INFO:teuthology.misc.health.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:31.982 INFO:teuthology.misc.health.smithi143.stderr: 2021-03-09T09:35:31.982 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 7 ==== mgrmap(e 4) v1 ==== 34858+0+0 (unknown 1337869971 0 0) 0x7f3d3c00d9f0 con 0x7f3d54099e70 2021-03-09T09:35:31.982 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.910+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 2796+0+0 (unknown 2970166609 0 0) 0x7f3d3c03a020 con 0x7f3d54099e70 2021-03-09T09:35:31.983 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.914+0000 7f3d592be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:31.983 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.914+0000 7f3d592be700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:31.983 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.914+0000 7f3d592be700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:31.983 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.914+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 --> v1:172.21.15.143:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2c004d50 con 0x7f3d54099e70 2021-03-09T09:35:31.984 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.914+0000 7f3d53fff700 10 monclient: get_auth_request con 0x7f3d3402fbb0 auth_method 0 2021-03-09T09:35:31.984 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.918+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+149001 (unknown 1092875540 0 1865811669) 0x7f3d3c0129e0 con 0x7f3d54099e70 2021-03-09T09:35:31.984 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.918+0000 7f3d3bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:31.984 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:31.918+0000 7f3d3bfff700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:32.076 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.074+0000 7f3d592be700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-03-09T09:35:32.076 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.074+0000 7f3d592be700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-03-09T09:35:32.076 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.074+0000 7f3d592be700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6791/0 2021-03-09T09:35:32.077 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.074+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 --> v1:172.21.15.143:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d2c005000 con 0x7f3d54099e70 2021-03-09T09:35:32.078 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.074+0000 7f3d3bfff700 1 -- 172.21.15.143:0/3904511274 <== mon.4 v1:172.21.15.143:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+6806 (unknown 238102980 0 2499871120) 0x7f3d3c0125d0 con 0x7f3d54099e70 2021-03-09T09:35:32.079 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.074+0000 7f3d3bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-03-09T09:35:32.079 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.074+0000 7f3d3bfff700 10 monclient: _finish_command 2 = system:0 2021-03-09T09:35:32.080 INFO:teuthology.misc.health.smithi143.stdout: 2021-03-09T09:35:32.080 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"8a34d6d4-5d27-4c75-9119-8e930fb2bed7","created":"2021-03-09T09:35:21.903074+0000","modified":"2021-03-09T09:35:31.121606+0000","last_up_change":"2021-03-09T09:35:31.121606+0000","last_in_change":"2021-03-09T09:35:24.182490+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"72ccc8f8-8bb0-4230-bfea-c47a8dadaf1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6800","nonce":16733}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6801","nonce":16733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6803","nonce":16733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6802","nonce":16733}]},"public_addr":"172.21.15.143:6800/16733","cluster_addr":"172.21.15.143:6801/16733","heartbeat_back_addr":"172.21.15.143:6803/16733","heartbeat_front_addr":"172.21.15.143:6802/16733","state":["exists","up"]},{"osd":1,"uuid":"dd598087-2308-4d24-9bbc-a6213e89ca6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6805","nonce":16734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6807","nonce":16734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6811","nonce":16734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6809","nonce":16734}]},"public_addr":"172.21.15.143:6805/16734","cluster_addr":"172.21.15.143:6807/16734","heartbeat_back_addr":"172.21.15.143:6811/16734","heartbeat_front_addr":"172.21.15.143:6809/16734","state":["exists","up"]},{"osd":2,"uuid":"a8112181-081d-43e4-af54-f8388903c61a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6804","nonce":16732}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6806","nonce":16732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6810","nonce":16732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.143:6808","nonce":16732}]},"public_addr":"172.21.15.143:6804/16732","cluster_addr":"172.21.15.143:6806/16732","heartbeat_back_addr":"172.21.15.143:6810/16732","heartbeat_front_addr":"172.21.15.143:6808/16732","state":["exists","up"]},{"osd":3,"uuid":"23b46e8e-3d64-48b2-be36-1594df839111","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6800","nonce":16821}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6801","nonce":16821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6803","nonce":16821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6802","nonce":16821}]},"public_addr":"172.21.15.173:6800/16821","cluster_addr":"172.21.15.173:6801/16821","heartbeat_back_addr":"172.21.15.173:6803/16821","heartbeat_front_addr":"172.21.15.173:6802/16821","state":["exists","up"]},{"osd":4,"uuid":"e50766be-3a46-4493-bb02-6eba34d6437d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6804","nonce":16822}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6805","nonce":16822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6807","nonce":16822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6806","nonce":16822}]},"public_addr":"172.21.15.173:6804/16822","cluster_addr":"172.21.15.173:6805/16822","heartbeat_back_addr":"172.21.15.173:6807/16822","heartbeat_front_addr":"172.21.15.173:6806/16822","state":["exists","up"]},{"osd":5,"uuid":"1169fffe-55ed-4cf1-b718-b9bf3d9ffe2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6808","nonce":16823}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6809","nonce":16823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6811","nonce":16823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.173:6810","nonce":16823}]},"public_addr":"172.21.15.173:6808/16823","cluster_addr":"172.21.15.173:6809/16823","heartbeat_back_addr":"172.21.15.173:6811/16823","heartbeat_front_addr":"172.21.15.173:6810/16823","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-09T09:35:32.082 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.078+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 >> v1:172.21.15.173:6812/16455 conn(0x7f3d3402fbb0 legacy=0x7f3d34032070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:32.082 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.078+0000 7f3d592be700 10 monclient: shutdown 2021-03-09T09:35:32.082 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.078+0000 7f3d592be700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:32.082 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.078+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 >> v1:172.21.15.143:6791/0 conn(0x7f3d54099e70 legacy=0x7f3d5413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:32.083 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.078+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 shutdown_connections 2021-03-09T09:35:32.083 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.078+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 >> 172.21.15.143:0/3904511274 conn(0x7f3d5406a240 msgr2=0x7f3d5412a120 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:32.083 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.078+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 shutdown_connections 2021-03-09T09:35:32.083 INFO:teuthology.misc.health.smithi143.stderr:2021-03-09T09:35:32.078+0000 7f3d592be700 1 -- 172.21.15.143:0/3904511274 wait complete. 2021-03-09T09:35:32.096 DEBUG:teuthology.misc:6 of 6 OSDs are up 2021-03-09T09:35:32.096 INFO:tasks.ceph:Creating RBD pool 2021-03-09T09:35:32.097 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd pool create rbd 8 2021-03-09T09:35:32.171 INFO:tasks.ceph.mon.a.smithi143.stderr:daemon-helper: command crashed with signal 6 2021-03-09T09:35:32.281 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient: get_monmap_and_config 2021-03-09T09:35:32.282 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient: build_initial_monmap 2021-03-09T09:35:32.282 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient: monmap: 2021-03-09T09:35:32.282 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2021-03-09T09:35:32.282 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:32.282 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:35:32.284918+0000 2021-03-09T09:35:32.283 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:35:32.284918+0000 2021-03-09T09:35:32.283 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:32.283 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:32.283 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:32.284 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:32.284 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:32.284 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:32.284 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:32.284 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:32.285 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:32.285 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:32.285 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:32.286 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:32.295 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 1 Processor -- start 2021-03-09T09:35:32.295 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 1 -- start start 2021-03-09T09:35:32.295 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient: init 2021-03-09T09:35:32.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:32.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient: _add_conns ranks=[5,1,7,2,0,6,3,4,8] 2021-03-09T09:35:32.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient(hunting): picked mon.noname-h con 0x7ff374095870 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:32.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient(hunting): picked mon.noname-f con 0x7ff374095c50 addr v1:172.21.15.173:6789/0 2021-03-09T09:35:32.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient(hunting): picked mon.noname-i con 0x7ff37412dce0 addr v1:172.21.15.173:6792/0 2021-03-09T09:35:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3740460b0 con 0x7ff374095c50 2021-03-09T09:35:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3740911b0 con 0x7ff374095870 2021-03-09T09:35:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 1 -- --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3740909d0 con 0x7ff37412dce0 2021-03-09T09:35:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff378d46700 10 monclient(hunting): authenticate will time out at 721.691723s 2021-03-09T09:35:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff3737fe700 1 --1- >> v1:172.21.15.173:6791/0 conn(0x7ff374095870 0x7ff3740917e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6791/0 says I am v1:172.21.15.143:60814/0 (socket says 172.21.15.143:60814) 2021-03-09T09:35:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.282+0000 7ff3737fe700 1 -- 172.21.15.143:0/2995148421 learned_addr learned my addr 172.21.15.143:0/2995148421 (peer_addr_for_me v1:172.21.15.143:0/0) 2021-03-09T09:35:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519688820 0 0) 0x7ff3740911b0 con 0x7ff374095870 2021-03-09T09:35:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient(hunting): my global_id is 4289 2021-03-09T09:35:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff358003230 con 0x7ff374095870 2021-03-09T09:35:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 <== mon.7 v1:172.21.15.173:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751259964 0 0) 0x7ff3740909d0 con 0x7ff37412dce0 2021-03-09T09:35:32.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:32.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient(hunting): my global_id is 4210 2021-03-09T09:35:32.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 --> v1:172.21.15.173:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3740911b0 con 0x7ff37412dce0 2021-03-09T09:35:32.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027107562 0 0) 0x7ff3740460b0 con 0x7ff374095c50 2021-03-09T09:35:32.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:32.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient(hunting): my global_id is 4222 2021-03-09T09:35:32.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3740909d0 con 0x7ff374095c50 2021-03-09T09:35:32.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 <== mon.5 v1:172.21.15.173:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1915216275 0 0) 0x7ff358003230 con 0x7ff374095870 2021-03-09T09:35:32.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 >> v1:172.21.15.173:6792/0 conn(0x7ff37412dce0 legacy=0x7ff3741300d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:32.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 >> v1:172.21.15.173:6789/0 conn(0x7ff374095c50 legacy=0x7ff374096090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:32.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: _finish_hunting 0 2021-03-09T09:35:32.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 monclient: found mon.noname-h 2021-03-09T09:35:32.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:32.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.173:6791/0 2021-03-09T09:35:32.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 --> v1:172.21.15.173:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff374090c70 con 0x7ff374095870 2021-03-09T09:35:32.304 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: _finish_auth 0 2021-03-09T09:35:32.304 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:32.304 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 5 monclient: authenticate success, global_id 4289 2021-03-09T09:35:32.305 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:32.305 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 <== mon.5 v1:172.21.15.173:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7ff360001820 con 0x7ff374095870 2021-03-09T09:35:32.305 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:32.305 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2021-03-09T09:35:32.306 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: dump: 2021-03-09T09:35:32.306 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2021-03-09T09:35:32.306 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:32.306 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.307 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.307 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:32.307 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:32.307 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:32.308 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:32.308 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:32.308 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:32.308 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:32.309 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:32.309 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:32.309 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:32.310 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:32.310 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:32.310 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T09:35:32.310 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 <== mon.5 v1:172.21.15.173:6791/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff360001a90 con 0x7ff374095870 2021-03-09T09:35:32.311 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:32.311 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 1 -- 172.21.15.143:0/2995148421 <== mon.5 v1:172.21.15.173:6791/0 5 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7ff3600048f0 con 0x7ff374095870 2021-03-09T09:35:32.311 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient: get_monmap_and_config success 2021-03-09T09:35:32.312 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 4 set_mon_vals no callback set 2021-03-09T09:35:32.312 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:32.312 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: got monmap 1 from mon.h (according to old e1) 2021-03-09T09:35:32.312 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3727fc700 10 monclient: dump: 2021-03-09T09:35:32.313 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2021-03-09T09:35:32.313 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:32.313 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.314 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.314 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:32.314 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:32.314 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:32.315 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:32.315 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:32.315 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:32.316 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:32.316 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:32.316 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:32.316 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:32.317 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:32.317 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:32.317 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient: shutdown 2021-03-09T09:35:32.317 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T09:35:32.318 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 -- 172.21.15.143:0/2995148421 >> v1:172.21.15.173:6791/0 conn(0x7ff374095870 legacy=0x7ff3740917e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:32.318 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 -- 172.21.15.143:0/2995148421 shutdown_connections 2021-03-09T09:35:32.318 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 -- 172.21.15.143:0/2995148421 >> 172.21.15.143:0/2995148421 conn(0x7ff37409ae90 msgr2=0x7ff37409b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T09:35:32.319 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 -- 172.21.15.143:0/2995148421 shutdown_connections 2021-03-09T09:35:32.319 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 -- 172.21.15.143:0/2995148421 wait complete. 2021-03-09T09:35:32.319 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient: build_initial_monmap 2021-03-09T09:35:32.320 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient: monmap: 2021-03-09T09:35:32.320 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2021-03-09T09:35:32.320 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T09:35:32.320 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2021-03-09T09:35:32.320 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2021-03-09T09:35:32.321 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2021-03-09T09:35:32.321 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:32.321 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2021-03-09T09:35:32.321 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.noname-f 2021-03-09T09:35:32.321 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2021-03-09T09:35:32.322 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.noname-g 2021-03-09T09:35:32.322 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.noname-c 2021-03-09T09:35:32.322 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.noname-h 2021-03-09T09:35:32.322 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.noname-d 2021-03-09T09:35:32.322 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.noname-i 2021-03-09T09:35:32.323 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.noname-e 2021-03-09T09:35:32.323 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:32.323 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 Processor -- start 2021-03-09T09:35:32.323 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 -- start start 2021-03-09T09:35:32.324 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient: init 2021-03-09T09:35:32.324 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:32.324 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient: _add_conns ranks=[0,7,5,8,6,1,3,4,2] 2021-03-09T09:35:32.324 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient(hunting): picked mon.noname-a con 0x7ff374095870 addr v1:172.21.15.143:6789/0 2021-03-09T09:35:32.324 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient(hunting): picked mon.noname-i con 0x7ff374095c50 addr v1:172.21.15.173:6792/0 2021-03-09T09:35:32.325 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient(hunting): picked mon.noname-h con 0x7ff37412dce0 addr v1:172.21.15.173:6791/0 2021-03-09T09:35:32.325 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37413b3e0 con 0x7ff374095870 2021-03-09T09:35:32.325 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff3737fe700 1 -- >> v1:172.21.15.143:6789/0 conn(0x7ff374095870 legacy=0x7ff37413a5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:35:32.325 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 -- --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff374067730 con 0x7ff37412dce0 2021-03-09T09:35:32.326 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 1 -- --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff374066b90 con 0x7ff374095c50 2021-03-09T09:35:32.326 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:32.326 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.286+0000 7ff378d46700 10 monclient(hunting): authenticate will time out at 721.695850s 2021-03-09T09:35:32.327 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff373fff700 1 --1- >> v1:172.21.15.173:6791/0 conn(0x7ff37412dce0 0x7ff374142b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6791/0 says I am v1:172.21.15.143:60824/0 (socket says 172.21.15.143:60824) 2021-03-09T09:35:32.327 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff372ffd700 1 --1- >> v1:172.21.15.173:6792/0 conn(0x7ff374095c50 0x7ff37413acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.173:6792/0 says I am v1:172.21.15.143:41112/0 (socket says 172.21.15.143:41112) 2021-03-09T09:35:32.327 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 learned_addr learned my addr 172.21.15.143:0/2492024602 (peer_addr_for_me v1:172.21.15.143:0/0) 2021-03-09T09:35:32.327 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.5 v1:172.21.15.173:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49401289 0 0) 0x7ff374067730 con 0x7ff37412dce0 2021-03-09T09:35:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): my global_id is 4298 2021-03-09T09:35:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff354003230 con 0x7ff37412dce0 2021-03-09T09:35:32.328 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.7 v1:172.21.15.173:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303211886 0 0) 0x7ff374066b90 con 0x7ff374095c50 2021-03-09T09:35:32.329 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:32.329 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): my global_id is 4219 2021-03-09T09:35:32.329 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff374067730 con 0x7ff374095c50 2021-03-09T09:35:32.329 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.5 v1:172.21.15.173:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1332404459 0 0) 0x7ff354003230 con 0x7ff37412dce0 2021-03-09T09:35:32.329 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff374066b90 con 0x7ff37412dce0 2021-03-09T09:35:32.330 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.5 v1:172.21.15.173:6791/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7ff368002db0 con 0x7ff37412dce0 2021-03-09T09:35:32.330 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:32.330 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2021-03-09T09:35:32.330 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): dump: 2021-03-09T09:35:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2021-03-09T09:35:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.331 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:32.332 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:32.333 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:32.334 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:32.334 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): _finish_auth 0 2021-03-09T09:35:32.334 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.7 v1:172.21.15.173:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 971022469 0 0) 0x7ff374067730 con 0x7ff374095c50 2021-03-09T09:35:32.334 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff354003230 con 0x7ff374095c50 2021-03-09T09:35:32.334 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.7 v1:172.21.15.173:6792/0 3 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7ff364002b10 con 0x7ff374095c50 2021-03-09T09:35:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2021-03-09T09:35:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient(hunting): dump: 2021-03-09T09:35:32.335 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2021-03-09T09:35:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.336 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:32.337 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:32.337 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:32.337 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:32.337 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:32.337 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:32.338 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:32.338 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:32.338 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:32.338 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:32.339 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:32.339 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:32.339 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.5 v1:172.21.15.173:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1566239343 0 0) 0x7ff374066b90 con 0x7ff37412dce0 2021-03-09T09:35:32.340 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff374095c50 legacy=0x7ff37413acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:35:32.340 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff374095870 legacy=0x7ff37413a5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:32.340 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient: _finish_hunting 0 2021-03-09T09:35:32.340 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 monclient: found mon.h 2021-03-09T09:35:32.341 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:32.341 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:32.341 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740660a0 con 0x7ff37412dce0 2021-03-09T09:35:32.342 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient: _finish_auth 0 2021-03-09T09:35:32.342 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:32.342 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 5 monclient: authenticate success, global_id 4298 2021-03-09T09:35:32.343 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 10 monclient: _renew_subs 2021-03-09T09:35:32.343 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:32.343 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff374067da0 con 0x7ff37412dce0 2021-03-09T09:35:32.343 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 10 monclient: _renew_subs 2021-03-09T09:35:32.344 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:32.344 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37412c010 con 0x7ff37412dce0 2021-03-09T09:35:32.344 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.5 v1:172.21.15.173:6791/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3680017c0 con 0x7ff37412dce0 2021-03-09T09:35:32.344 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:32.345 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.5 v1:172.21.15.173:6791/0 6 ==== mon_map magic: 0 v1 ==== 722+0+0 (unknown 2049694188 0 0) 0x7ff368001cf0 con 0x7ff37412dce0 2021-03-09T09:35:32.345 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:32.345 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient: got monmap 1 from mon.h (according to old e1) 2021-03-09T09:35:32.345 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff371ffb700 4 set_mon_vals no callback set 2021-03-09T09:35:32.346 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 10 monclient: dump: 2021-03-09T09:35:32.346 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2021-03-09T09:35:32.346 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 8a34d6d4-5d27-4c75-9119-8e930fb2bed7 2021-03-09T09:35:32.346 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.346 INFO:teuthology.orchestra.run.smithi143.stderr:created 2021-03-09T09:34:44.621015+0000 2021-03-09T09:35:32.347 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:32.347 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2021-03-09T09:35:32.347 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2021-03-09T09:35:32.347 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.173:6789/0 mon.f 2021-03-09T09:35:32.347 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.b 2021-03-09T09:35:32.348 INFO:teuthology.orchestra.run.smithi143.stderr:3: v1:172.21.15.173:6790/0 mon.g 2021-03-09T09:35:32.348 INFO:teuthology.orchestra.run.smithi143.stderr:4: v1:172.21.15.143:6791/0 mon.c 2021-03-09T09:35:32.348 INFO:teuthology.orchestra.run.smithi143.stderr:5: v1:172.21.15.173:6791/0 mon.h 2021-03-09T09:35:32.348 INFO:teuthology.orchestra.run.smithi143.stderr:6: v1:172.21.15.143:6792/0 mon.d 2021-03-09T09:35:32.349 INFO:teuthology.orchestra.run.smithi143.stderr:7: v1:172.21.15.173:6792/0 mon.i 2021-03-09T09:35:32.349 INFO:teuthology.orchestra.run.smithi143.stderr:8: v1:172.21.15.143:6793/0 mon.e 2021-03-09T09:35:32.349 INFO:teuthology.orchestra.run.smithi143.stderr: 2021-03-09T09:35:32.349 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.5 v1:172.21.15.173:6791/0 7 ==== mgrmap(e 4) v1 ==== 34858+0+0 (unknown 1337869971 0 0) 0x7ff368001ed0 con 0x7ff37412dce0 2021-03-09T09:35:32.349 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.5 v1:172.21.15.173:6791/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 2796+0+0 (unknown 2970166609 0 0) 0x7ff368039fd0 con 0x7ff37412dce0 2021-03-09T09:35:32.350 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:32.350 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:32.350 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:32.350 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.290+0000 7ff378d46700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34c004d60 con 0x7ff37412dce0 2021-03-09T09:35:32.351 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.294+0000 7ff3737fe700 10 monclient: get_auth_request con 0x7ff354030290 auth_method 0 2021-03-09T09:35:32.351 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.302+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 <== mon.5 v1:172.21.15.173:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+149001 (unknown 1092875540 0 1865811669) 0x7ff368012730 con 0x7ff37412dce0 2021-03-09T09:35:32.351 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.302+0000 7ff35f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T09:35:32.352 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.302+0000 7ff35f7fe700 10 monclient: _finish_command 1 = system:0 2021-03-09T09:35:32.460 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.458+0000 7ff378d46700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:32.460 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.458+0000 7ff378d46700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:32.461 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.458+0000 7ff378d46700 10 monclient: _send_mon_message to mon.h at v1:172.21.15.173:6791/0 2021-03-09T09:35:32.461 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:32.458+0000 7ff378d46700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7ff34c005590 con 0x7ff37412dce0 2021-03-09T09:35:35.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:35.286+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:35.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:35.286+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:35.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:35.286+0000 7ff35e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:36.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:36.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:36.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:36.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:37.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:37.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:37.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:37.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:38.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:38.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:38.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:38.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:39.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:39.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:39.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:39.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:40.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:40.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:40.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:40.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:41.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:41.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:41.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:41.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:42.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:42.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:42.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:42.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:43.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:43.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:43.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:43.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:44.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:44.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:44.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:44.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:45.289 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:45.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:45.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:45.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:45.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:45.290+0000 7ff35e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:46.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:46.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:46.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:46.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:47.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:47.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:47.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:47.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:48.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:48.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:48.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:48.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:49.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:49.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:49.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:49.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:50.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:50.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:50.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:50.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:51.221 INFO:tasks.ceph.mon.f.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f0f143c8700 time 2021-03-09T09:35:51.225131+0000 2021-03-09T09:35:51.222 INFO:tasks.ceph.mon.f.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:35:51.235 INFO:tasks.ceph.mon.f.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:51.235 INFO:tasks.ceph.mon.f.smithi173.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f0f20b8a3dd] 2021-03-09T09:35:51.235 INFO:tasks.ceph.mon.f.smithi173.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0f20b8a5b8] 2021-03-09T09:35:51.236 INFO:tasks.ceph.mon.f.smithi173.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x563e7423da65] 2021-03-09T09:35:51.236 INFO:tasks.ceph.mon.f.smithi173.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x563e74289a71] 2021-03-09T09:35:51.236 INFO:tasks.ceph.mon.f.smithi173.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x563e742b9295] 2021-03-09T09:35:51.236 INFO:tasks.ceph.mon.f.smithi173.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x563e742c124b] 2021-03-09T09:35:51.237 INFO:tasks.ceph.mon.f.smithi173.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x563e742c17ab] 2021-03-09T09:35:51.237 INFO:tasks.ceph.mon.f.smithi173.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x563e74230ade] 2021-03-09T09:35:51.237 INFO:tasks.ceph.mon.f.smithi173.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e74186134] 2021-03-09T09:35:51.237 INFO:tasks.ceph.mon.f.smithi173.stderr: 10: (C_MonOp::finish(int)+0x45) [0x563e74124aa5] 2021-03-09T09:35:51.238 INFO:tasks.ceph.mon.f.smithi173.stderr: 11: (Context::complete(int)+0x9) [0x563e74122059] 2021-03-09T09:35:51.238 INFO:tasks.ceph.mon.f.smithi173.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x563e7414d548] 2021-03-09T09:35:51.238 INFO:tasks.ceph.mon.f.smithi173.stderr: 13: (Paxos::finish_round()+0x9b) [0x563e742287db] 2021-03-09T09:35:51.238 INFO:tasks.ceph.mon.f.smithi173.stderr: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x563e74229a06] 2021-03-09T09:35:51.239 INFO:tasks.ceph.mon.f.smithi173.stderr: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x563e7422a633] 2021-03-09T09:35:51.239 INFO:tasks.ceph.mon.f.smithi173.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x563e7411fde1] 2021-03-09T09:35:51.239 INFO:tasks.ceph.mon.f.smithi173.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x563e7412066a] 2021-03-09T09:35:51.239 INFO:tasks.ceph.mon.f.smithi173.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x563e7414e178] 2021-03-09T09:35:51.240 INFO:tasks.ceph.mon.f.smithi173.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f0f20d8c2ca] 2021-03-09T09:35:51.240 INFO:tasks.ceph.mon.f.smithi173.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0f20e3671d] 2021-03-09T09:35:51.240 INFO:tasks.ceph.mon.f.smithi173.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0f1f9fa6db] 2021-03-09T09:35:51.240 INFO:tasks.ceph.mon.f.smithi173.stderr: 22: clone() 2021-03-09T09:35:51.240 INFO:tasks.ceph.mon.f.smithi173.stderr:*** Caught signal (Aborted) ** 2021-03-09T09:35:51.241 INFO:tasks.ceph.mon.f.smithi173.stderr: in thread 7f0f143c8700 thread_name:ms_dispatch 2021-03-09T09:35:51.241 INFO:tasks.ceph.mon.f.smithi173.stderr:2021-03-09T09:35:51.233+0000 7f0f143c8700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f0f143c8700 time 2021-03-09T09:35:51.225131+0000 2021-03-09T09:35:51.241 INFO:tasks.ceph.mon.f.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:35:51.242 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.242 INFO:tasks.ceph.mon.f.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:51.242 INFO:tasks.ceph.mon.f.smithi173.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f0f20b8a3dd] 2021-03-09T09:35:51.242 INFO:tasks.ceph.mon.f.smithi173.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0f20b8a5b8] 2021-03-09T09:35:51.242 INFO:tasks.ceph.mon.f.smithi173.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x563e7423da65] 2021-03-09T09:35:51.243 INFO:tasks.ceph.mon.f.smithi173.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x563e74289a71] 2021-03-09T09:35:51.243 INFO:tasks.ceph.mon.f.smithi173.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x563e742b9295] 2021-03-09T09:35:51.243 INFO:tasks.ceph.mon.f.smithi173.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x563e742c124b] 2021-03-09T09:35:51.243 INFO:tasks.ceph.mon.f.smithi173.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x563e742c17ab] 2021-03-09T09:35:51.244 INFO:tasks.ceph.mon.f.smithi173.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x563e74230ade] 2021-03-09T09:35:51.244 INFO:tasks.ceph.mon.f.smithi173.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e74186134] 2021-03-09T09:35:51.244 INFO:tasks.ceph.mon.f.smithi173.stderr: 10: (C_MonOp::finish(int)+0x45) [0x563e74124aa5] 2021-03-09T09:35:51.244 INFO:tasks.ceph.mon.f.smithi173.stderr: 11: (Context::complete(int)+0x9) [0x563e74122059] 2021-03-09T09:35:51.245 INFO:tasks.ceph.mon.f.smithi173.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x563e7414d548] 2021-03-09T09:35:51.245 INFO:tasks.ceph.mon.f.smithi173.stderr: 13: (Paxos::finish_round()+0x9b) [0x563e742287db] 2021-03-09T09:35:51.245 INFO:tasks.ceph.mon.f.smithi173.stderr: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x563e74229a06] 2021-03-09T09:35:51.245 INFO:tasks.ceph.mon.f.smithi173.stderr: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x563e7422a633] 2021-03-09T09:35:51.246 INFO:tasks.ceph.mon.f.smithi173.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x563e7411fde1] 2021-03-09T09:35:51.246 INFO:tasks.ceph.mon.f.smithi173.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x563e7412066a] 2021-03-09T09:35:51.246 INFO:tasks.ceph.mon.f.smithi173.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x563e7414e178] 2021-03-09T09:35:51.246 INFO:tasks.ceph.mon.f.smithi173.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f0f20d8c2ca] 2021-03-09T09:35:51.246 INFO:tasks.ceph.mon.f.smithi173.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0f20e3671d] 2021-03-09T09:35:51.247 INFO:tasks.ceph.mon.f.smithi173.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0f1f9fa6db] 2021-03-09T09:35:51.247 INFO:tasks.ceph.mon.f.smithi173.stderr: 22: clone() 2021-03-09T09:35:51.247 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.249 INFO:tasks.ceph.mon.f.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:51.250 INFO:tasks.ceph.mon.f.smithi173.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f0f1fa05980] 2021-03-09T09:35:51.250 INFO:tasks.ceph.mon.f.smithi173.stderr: 2: gsignal() 2021-03-09T09:35:51.251 INFO:tasks.ceph.mon.f.smithi173.stderr: 3: abort() 2021-03-09T09:35:51.252 INFO:tasks.ceph.mon.f.smithi173.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f0f20b8a42e] 2021-03-09T09:35:51.252 INFO:tasks.ceph.mon.f.smithi173.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0f20b8a5b8] 2021-03-09T09:35:51.253 INFO:tasks.ceph.mon.f.smithi173.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x563e7423da65] 2021-03-09T09:35:51.253 INFO:tasks.ceph.mon.f.smithi173.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x563e74289a71] 2021-03-09T09:35:51.253 INFO:tasks.ceph.mon.f.smithi173.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x563e742b9295] 2021-03-09T09:35:51.253 INFO:tasks.ceph.mon.f.smithi173.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x563e742c124b] 2021-03-09T09:35:51.254 INFO:tasks.ceph.mon.f.smithi173.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x563e742c17ab] 2021-03-09T09:35:51.254 INFO:tasks.ceph.mon.f.smithi173.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x563e74230ade] 2021-03-09T09:35:51.254 INFO:tasks.ceph.mon.f.smithi173.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e74186134] 2021-03-09T09:35:51.254 INFO:tasks.ceph.mon.f.smithi173.stderr: 13: (C_MonOp::finish(int)+0x45) [0x563e74124aa5] 2021-03-09T09:35:51.255 INFO:tasks.ceph.mon.f.smithi173.stderr: 14: (Context::complete(int)+0x9) [0x563e74122059] 2021-03-09T09:35:51.255 INFO:tasks.ceph.mon.f.smithi173.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x563e7414d548] 2021-03-09T09:35:51.255 INFO:tasks.ceph.mon.f.smithi173.stderr: 16: (Paxos::finish_round()+0x9b) [0x563e742287db] 2021-03-09T09:35:51.255 INFO:tasks.ceph.mon.f.smithi173.stderr: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x563e74229a06] 2021-03-09T09:35:51.255 INFO:tasks.ceph.mon.f.smithi173.stderr: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x563e7422a633] 2021-03-09T09:35:51.256 INFO:tasks.ceph.mon.f.smithi173.stderr: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x563e7411fde1] 2021-03-09T09:35:51.256 INFO:tasks.ceph.mon.f.smithi173.stderr: 20: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x563e7412066a] 2021-03-09T09:35:51.256 INFO:tasks.ceph.mon.f.smithi173.stderr: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x563e7414e178] 2021-03-09T09:35:51.256 INFO:tasks.ceph.mon.f.smithi173.stderr: 22: (DispatchQueue::entry()+0x11ea) [0x7f0f20d8c2ca] 2021-03-09T09:35:51.257 INFO:tasks.ceph.mon.f.smithi173.stderr: 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0f20e3671d] 2021-03-09T09:35:51.257 INFO:tasks.ceph.mon.f.smithi173.stderr: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0f1f9fa6db] 2021-03-09T09:35:51.257 INFO:tasks.ceph.mon.f.smithi173.stderr: 25: clone() 2021-03-09T09:35:51.257 INFO:tasks.ceph.mon.f.smithi173.stderr:2021-03-09T09:35:51.241+0000 7f0f143c8700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:35:51.257 INFO:tasks.ceph.mon.f.smithi173.stderr: in thread 7f0f143c8700 thread_name:ms_dispatch 2021-03-09T09:35:51.258 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.258 INFO:tasks.ceph.mon.f.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:51.258 INFO:tasks.ceph.mon.f.smithi173.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f0f1fa05980] 2021-03-09T09:35:51.258 INFO:tasks.ceph.mon.f.smithi173.stderr: 2: gsignal() 2021-03-09T09:35:51.259 INFO:tasks.ceph.mon.f.smithi173.stderr: 3: abort() 2021-03-09T09:35:51.259 INFO:tasks.ceph.mon.f.smithi173.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f0f20b8a42e] 2021-03-09T09:35:51.259 INFO:tasks.ceph.mon.f.smithi173.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0f20b8a5b8] 2021-03-09T09:35:51.259 INFO:tasks.ceph.mon.f.smithi173.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x563e7423da65] 2021-03-09T09:35:51.259 INFO:tasks.ceph.mon.f.smithi173.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x563e74289a71] 2021-03-09T09:35:51.260 INFO:tasks.ceph.mon.f.smithi173.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x563e742b9295] 2021-03-09T09:35:51.260 INFO:tasks.ceph.mon.f.smithi173.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x563e742c124b] 2021-03-09T09:35:51.261 INFO:tasks.ceph.mon.f.smithi173.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x563e742c17ab] 2021-03-09T09:35:51.261 INFO:tasks.ceph.mon.f.smithi173.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x563e74230ade] 2021-03-09T09:35:51.261 INFO:tasks.ceph.mon.f.smithi173.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e74186134] 2021-03-09T09:35:51.261 INFO:tasks.ceph.mon.f.smithi173.stderr: 13: (C_MonOp::finish(int)+0x45) [0x563e74124aa5] 2021-03-09T09:35:51.262 INFO:tasks.ceph.mon.f.smithi173.stderr: 14: (Context::complete(int)+0x9) [0x563e74122059] 2021-03-09T09:35:51.262 INFO:tasks.ceph.mon.f.smithi173.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x563e7414d548] 2021-03-09T09:35:51.262 INFO:tasks.ceph.mon.f.smithi173.stderr: 16: (Paxos::finish_round()+0x9b) [0x563e742287db] 2021-03-09T09:35:51.262 INFO:tasks.ceph.mon.f.smithi173.stderr: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x563e74229a06] 2021-03-09T09:35:51.263 INFO:tasks.ceph.mon.f.smithi173.stderr: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x563e7422a633] 2021-03-09T09:35:51.263 INFO:tasks.ceph.mon.f.smithi173.stderr: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x563e7411fde1] 2021-03-09T09:35:51.263 INFO:tasks.ceph.mon.f.smithi173.stderr: 20: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x563e7412066a] 2021-03-09T09:35:51.263 INFO:tasks.ceph.mon.f.smithi173.stderr: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x563e7414e178] 2021-03-09T09:35:51.264 INFO:tasks.ceph.mon.f.smithi173.stderr: 22: (DispatchQueue::entry()+0x11ea) [0x7f0f20d8c2ca] 2021-03-09T09:35:51.264 INFO:tasks.ceph.mon.f.smithi173.stderr: 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0f20e3671d] 2021-03-09T09:35:51.264 INFO:tasks.ceph.mon.f.smithi173.stderr: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0f1f9fa6db] 2021-03-09T09:35:51.264 INFO:tasks.ceph.mon.f.smithi173.stderr: 25: clone() 2021-03-09T09:35:51.265 INFO:tasks.ceph.mon.f.smithi173.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:35:51.265 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.266 INFO:tasks.ceph.mon.f.smithi173.stderr: -1> 2021-03-09T09:35:51.233+0000 7f0f143c8700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f0f143c8700 time 2021-03-09T09:35:51.225131+0000 2021-03-09T09:35:51.266 INFO:tasks.ceph.mon.f.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:35:51.266 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.267 INFO:tasks.ceph.mon.f.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:51.268 INFO:tasks.ceph.mon.f.smithi173.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f0f20b8a3dd] 2021-03-09T09:35:51.268 INFO:tasks.ceph.mon.f.smithi173.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0f20b8a5b8] 2021-03-09T09:35:51.268 INFO:tasks.ceph.mon.f.smithi173.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x563e7423da65] 2021-03-09T09:35:51.269 INFO:tasks.ceph.mon.f.smithi173.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x563e74289a71] 2021-03-09T09:35:51.269 INFO:tasks.ceph.mon.f.smithi173.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x563e742b9295] 2021-03-09T09:35:51.270 INFO:tasks.ceph.mon.f.smithi173.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x563e742c124b] 2021-03-09T09:35:51.270 INFO:tasks.ceph.mon.f.smithi173.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x563e742c17ab] 2021-03-09T09:35:51.271 INFO:tasks.ceph.mon.f.smithi173.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x563e74230ade] 2021-03-09T09:35:51.271 INFO:tasks.ceph.mon.f.smithi173.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e74186134] 2021-03-09T09:35:51.271 INFO:tasks.ceph.mon.f.smithi173.stderr: 10: (C_MonOp::finish(int)+0x45) [0x563e74124aa5] 2021-03-09T09:35:51.272 INFO:tasks.ceph.mon.f.smithi173.stderr: 11: (Context::complete(int)+0x9) [0x563e74122059] 2021-03-09T09:35:51.272 INFO:tasks.ceph.mon.f.smithi173.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x563e7414d548] 2021-03-09T09:35:51.273 INFO:tasks.ceph.mon.f.smithi173.stderr: 13: (Paxos::finish_round()+0x9b) [0x563e742287db] 2021-03-09T09:35:51.273 INFO:tasks.ceph.mon.f.smithi173.stderr: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x563e74229a06] 2021-03-09T09:35:51.273 INFO:tasks.ceph.mon.f.smithi173.stderr: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x563e7422a633] 2021-03-09T09:35:51.274 INFO:tasks.ceph.mon.f.smithi173.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x563e7411fde1] 2021-03-09T09:35:51.274 INFO:tasks.ceph.mon.f.smithi173.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x563e7412066a] 2021-03-09T09:35:51.275 INFO:tasks.ceph.mon.f.smithi173.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x563e7414e178] 2021-03-09T09:35:51.275 INFO:tasks.ceph.mon.f.smithi173.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f0f20d8c2ca] 2021-03-09T09:35:51.276 INFO:tasks.ceph.mon.f.smithi173.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0f20e3671d] 2021-03-09T09:35:51.276 INFO:tasks.ceph.mon.f.smithi173.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0f1f9fa6db] 2021-03-09T09:35:51.276 INFO:tasks.ceph.mon.f.smithi173.stderr: 22: clone() 2021-03-09T09:35:51.277 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.277 INFO:tasks.ceph.mon.f.smithi173.stderr: 0> 2021-03-09T09:35:51.241+0000 7f0f143c8700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:35:51.278 INFO:tasks.ceph.mon.f.smithi173.stderr: in thread 7f0f143c8700 thread_name:ms_dispatch 2021-03-09T09:35:51.278 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.279 INFO:tasks.ceph.mon.f.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:51.279 INFO:tasks.ceph.mon.f.smithi173.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f0f1fa05980] 2021-03-09T09:35:51.279 INFO:tasks.ceph.mon.f.smithi173.stderr: 2: gsignal() 2021-03-09T09:35:51.280 INFO:tasks.ceph.mon.f.smithi173.stderr: 3: abort() 2021-03-09T09:35:51.280 INFO:tasks.ceph.mon.f.smithi173.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f0f20b8a42e] 2021-03-09T09:35:51.281 INFO:tasks.ceph.mon.f.smithi173.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0f20b8a5b8] 2021-03-09T09:35:51.281 INFO:tasks.ceph.mon.f.smithi173.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x563e7423da65] 2021-03-09T09:35:51.282 INFO:tasks.ceph.mon.f.smithi173.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x563e74289a71] 2021-03-09T09:35:51.282 INFO:tasks.ceph.mon.f.smithi173.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x563e742b9295] 2021-03-09T09:35:51.282 INFO:tasks.ceph.mon.f.smithi173.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x563e742c124b] 2021-03-09T09:35:51.283 INFO:tasks.ceph.mon.f.smithi173.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x563e742c17ab] 2021-03-09T09:35:51.283 INFO:tasks.ceph.mon.f.smithi173.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x563e74230ade] 2021-03-09T09:35:51.284 INFO:tasks.ceph.mon.f.smithi173.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e74186134] 2021-03-09T09:35:51.284 INFO:tasks.ceph.mon.f.smithi173.stderr: 13: (C_MonOp::finish(int)+0x45) [0x563e74124aa5] 2021-03-09T09:35:51.285 INFO:tasks.ceph.mon.f.smithi173.stderr: 14: (Context::complete(int)+0x9) [0x563e74122059] 2021-03-09T09:35:51.285 INFO:tasks.ceph.mon.f.smithi173.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x563e7414d548] 2021-03-09T09:35:51.286 INFO:tasks.ceph.mon.f.smithi173.stderr: 16: (Paxos::finish_round()+0x9b) [0x563e742287db] 2021-03-09T09:35:51.286 INFO:tasks.ceph.mon.f.smithi173.stderr: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x563e74229a06] 2021-03-09T09:35:51.287 INFO:tasks.ceph.mon.f.smithi173.stderr: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x563e7422a633] 2021-03-09T09:35:51.287 INFO:tasks.ceph.mon.f.smithi173.stderr: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x563e7411fde1] 2021-03-09T09:35:51.287 INFO:tasks.ceph.mon.f.smithi173.stderr: 20: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x563e7412066a] 2021-03-09T09:35:51.288 INFO:tasks.ceph.mon.f.smithi173.stderr: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x563e7414e178] 2021-03-09T09:35:51.288 INFO:tasks.ceph.mon.f.smithi173.stderr: 22: (DispatchQueue::entry()+0x11ea) [0x7f0f20d8c2ca] 2021-03-09T09:35:51.289 INFO:tasks.ceph.mon.f.smithi173.stderr: 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0f20e3671d] 2021-03-09T09:35:51.289 INFO:tasks.ceph.mon.f.smithi173.stderr: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0f1f9fa6db] 2021-03-09T09:35:51.290 INFO:tasks.ceph.mon.f.smithi173.stderr: 25: clone() 2021-03-09T09:35:51.290 INFO:tasks.ceph.mon.f.smithi173.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:35:51.291 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:51.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:51.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:51.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:51.293 INFO:tasks.ceph.mon.f.smithi173.stderr: -1> 2021-03-09T09:35:51.233+0000 7f0f143c8700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f0f143c8700 time 2021-03-09T09:35:51.225131+0000 2021-03-09T09:35:51.293 INFO:tasks.ceph.mon.f.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:35:51.294 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.294 INFO:tasks.ceph.mon.f.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:51.295 INFO:tasks.ceph.mon.f.smithi173.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f0f20b8a3dd] 2021-03-09T09:35:51.295 INFO:tasks.ceph.mon.f.smithi173.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0f20b8a5b8] 2021-03-09T09:35:51.296 INFO:tasks.ceph.mon.f.smithi173.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x563e7423da65] 2021-03-09T09:35:51.296 INFO:tasks.ceph.mon.f.smithi173.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x563e74289a71] 2021-03-09T09:35:51.296 INFO:tasks.ceph.mon.f.smithi173.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x563e742b9295] 2021-03-09T09:35:51.297 INFO:tasks.ceph.mon.f.smithi173.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x563e742c124b] 2021-03-09T09:35:51.297 INFO:tasks.ceph.mon.f.smithi173.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x563e742c17ab] 2021-03-09T09:35:51.298 INFO:tasks.ceph.mon.f.smithi173.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x563e74230ade] 2021-03-09T09:35:51.298 INFO:tasks.ceph.mon.f.smithi173.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e74186134] 2021-03-09T09:35:51.298 INFO:tasks.ceph.mon.f.smithi173.stderr: 10: (C_MonOp::finish(int)+0x45) [0x563e74124aa5] 2021-03-09T09:35:51.299 INFO:tasks.ceph.mon.f.smithi173.stderr: 11: (Context::complete(int)+0x9) [0x563e74122059] 2021-03-09T09:35:51.299 INFO:tasks.ceph.mon.f.smithi173.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x563e7414d548] 2021-03-09T09:35:51.300 INFO:tasks.ceph.mon.f.smithi173.stderr: 13: (Paxos::finish_round()+0x9b) [0x563e742287db] 2021-03-09T09:35:51.300 INFO:tasks.ceph.mon.f.smithi173.stderr: 14: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x563e74229a06] 2021-03-09T09:35:51.300 INFO:tasks.ceph.mon.f.smithi173.stderr: 15: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x563e7422a633] 2021-03-09T09:35:51.301 INFO:tasks.ceph.mon.f.smithi173.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x563e7411fde1] 2021-03-09T09:35:51.301 INFO:tasks.ceph.mon.f.smithi173.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x563e7412066a] 2021-03-09T09:35:51.302 INFO:tasks.ceph.mon.f.smithi173.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x563e7414e178] 2021-03-09T09:35:51.302 INFO:tasks.ceph.mon.f.smithi173.stderr: 19: (DispatchQueue::entry()+0x11ea) [0x7f0f20d8c2ca] 2021-03-09T09:35:51.303 INFO:tasks.ceph.mon.f.smithi173.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0f20e3671d] 2021-03-09T09:35:51.303 INFO:tasks.ceph.mon.f.smithi173.stderr: 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0f1f9fa6db] 2021-03-09T09:35:51.303 INFO:tasks.ceph.mon.f.smithi173.stderr: 22: clone() 2021-03-09T09:35:51.304 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.304 INFO:tasks.ceph.mon.f.smithi173.stderr: 0> 2021-03-09T09:35:51.241+0000 7f0f143c8700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:35:51.305 INFO:tasks.ceph.mon.f.smithi173.stderr: in thread 7f0f143c8700 thread_name:ms_dispatch 2021-03-09T09:35:51.305 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:51.306 INFO:tasks.ceph.mon.f.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:35:51.306 INFO:tasks.ceph.mon.f.smithi173.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f0f1fa05980] 2021-03-09T09:35:51.307 INFO:tasks.ceph.mon.f.smithi173.stderr: 2: gsignal() 2021-03-09T09:35:51.307 INFO:tasks.ceph.mon.f.smithi173.stderr: 3: abort() 2021-03-09T09:35:51.307 INFO:tasks.ceph.mon.f.smithi173.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f0f20b8a42e] 2021-03-09T09:35:51.308 INFO:tasks.ceph.mon.f.smithi173.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0f20b8a5b8] 2021-03-09T09:35:51.308 INFO:tasks.ceph.mon.f.smithi173.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x563e7423da65] 2021-03-09T09:35:51.309 INFO:tasks.ceph.mon.f.smithi173.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x563e74289a71] 2021-03-09T09:35:51.309 INFO:tasks.ceph.mon.f.smithi173.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x563e742b9295] 2021-03-09T09:35:51.310 INFO:tasks.ceph.mon.f.smithi173.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x563e742c124b] 2021-03-09T09:35:51.310 INFO:tasks.ceph.mon.f.smithi173.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x563e742c17ab] 2021-03-09T09:35:51.310 INFO:tasks.ceph.mon.f.smithi173.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x563e74230ade] 2021-03-09T09:35:51.311 INFO:tasks.ceph.mon.f.smithi173.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x563e74186134] 2021-03-09T09:35:51.311 INFO:tasks.ceph.mon.f.smithi173.stderr: 13: (C_MonOp::finish(int)+0x45) [0x563e74124aa5] 2021-03-09T09:35:51.312 INFO:tasks.ceph.mon.f.smithi173.stderr: 14: (Context::complete(int)+0x9) [0x563e74122059] 2021-03-09T09:35:51.312 INFO:tasks.ceph.mon.f.smithi173.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x563e7414d548] 2021-03-09T09:35:51.312 INFO:tasks.ceph.mon.f.smithi173.stderr: 16: (Paxos::finish_round()+0x9b) [0x563e742287db] 2021-03-09T09:35:51.313 INFO:tasks.ceph.mon.f.smithi173.stderr: 17: (Paxos::handle_last(boost::intrusive_ptr)+0xc46) [0x563e74229a06] 2021-03-09T09:35:51.313 INFO:tasks.ceph.mon.f.smithi173.stderr: 18: (Paxos::dispatch(boost::intrusive_ptr)+0x3c3) [0x563e7422a633] 2021-03-09T09:35:51.314 INFO:tasks.ceph.mon.f.smithi173.stderr: 19: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1401) [0x563e7411fde1] 2021-03-09T09:35:51.314 INFO:tasks.ceph.mon.f.smithi173.stderr: 20: (Monitor::_ms_dispatch(Message*)+0x4ea) [0x563e7412066a] 2021-03-09T09:35:51.315 INFO:tasks.ceph.mon.f.smithi173.stderr: 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x58) [0x563e7414e178] 2021-03-09T09:35:51.315 INFO:tasks.ceph.mon.f.smithi173.stderr: 22: (DispatchQueue::entry()+0x11ea) [0x7f0f20d8c2ca] 2021-03-09T09:35:51.315 INFO:tasks.ceph.mon.f.smithi173.stderr: 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f0f20e3671d] 2021-03-09T09:35:51.316 INFO:tasks.ceph.mon.f.smithi173.stderr: 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f0f1f9fa6db] 2021-03-09T09:35:51.316 INFO:tasks.ceph.mon.f.smithi173.stderr: 25: clone() 2021-03-09T09:35:51.317 INFO:tasks.ceph.mon.f.smithi173.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:35:51.317 INFO:tasks.ceph.mon.f.smithi173.stderr: 2021-03-09T09:35:52.037 INFO:tasks.ceph.mon.f.smithi173.stderr:daemon-helper: command crashed with signal 6 2021-03-09T09:35:52.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:52.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:52.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:52.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:53.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:53.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:53.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:53.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:54.290 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:54.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:54.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:54.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:55.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:55.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:55.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:55.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:55.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:55.294+0000 7ff35e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:56.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:56.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:56.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:56.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:57.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:57.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:57.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:57.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:58.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:58.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:58.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:58.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:59.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:59.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:35:59.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:35:59.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:00.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:00.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:00.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:00.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:01.291 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:01.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:01.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:01.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:01.659 INFO:tasks.ceph.mon.b.smithi143.stderr:2021-03-09T09:36:01.662+0000 7fa985931700 -1 mon.b@2(electing) e1 get_health_metrics reporting 11 slow ops, oldest is mon_command({"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1} v 0) 2021-03-09T09:36:01.676 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:01.678+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 1 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:01.702 INFO:tasks.ceph.mon.g.smithi173.stderr:2021-03-09T09:36:01.705+0000 7f2a6dca2700 -1 mon.g@3(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:01.731 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:01.733+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:02.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:02.290+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:02.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:02.290+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:03.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:03.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:03.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:03.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:04.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:04.289+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:04.292 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:04.289+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:05.293 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:05.293+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:05.294 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:05.293+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:05.294 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:05.293+0000 7ff35e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:06.239 INFO:tasks.ceph.mon.b.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fa981929700 time 2021-03-09T09:36:06.242011+0000 2021-03-09T09:36:06.239 INFO:tasks.ceph.mon.b.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:06.249 INFO:tasks.ceph.mon.b.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:06.250 INFO:tasks.ceph.mon.b.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fa98f8ee3dd] 2021-03-09T09:36:06.250 INFO:tasks.ceph.mon.b.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fa98f8ee5b8] 2021-03-09T09:36:06.250 INFO:tasks.ceph.mon.b.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x561d96c45a65] 2021-03-09T09:36:06.251 INFO:tasks.ceph.mon.b.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x561d96c91a71] 2021-03-09T09:36:06.251 INFO:tasks.ceph.mon.b.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x561d96cc1295] 2021-03-09T09:36:06.251 INFO:tasks.ceph.mon.b.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x561d96cc924b] 2021-03-09T09:36:06.252 INFO:tasks.ceph.mon.b.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x561d96cc97ab] 2021-03-09T09:36:06.252 INFO:tasks.ceph.mon.b.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x561d96c38ade] 2021-03-09T09:36:06.252 INFO:tasks.ceph.mon.b.smithi143.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x561d96b8e134] 2021-03-09T09:36:06.253 INFO:tasks.ceph.mon.b.smithi143.stderr: 10: (C_MonOp::finish(int)+0x45) [0x561d96b2caa5] 2021-03-09T09:36:06.253 INFO:tasks.ceph.mon.b.smithi143.stderr: 11: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.253 INFO:tasks.ceph.mon.b.smithi143.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x561d96b55548] 2021-03-09T09:36:06.254 INFO:tasks.ceph.mon.b.smithi143.stderr: 13: (Paxos::finish_round()+0x9b) [0x561d96c307db] 2021-03-09T09:36:06.254 INFO:tasks.ceph.mon.b.smithi143.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x561d96c330c0] 2021-03-09T09:36:06.254 INFO:tasks.ceph.mon.b.smithi143.stderr: 15: (C_Committed::finish(int)+0x41) [0x561d96c36861] 2021-03-09T09:36:06.255 INFO:tasks.ceph.mon.b.smithi143.stderr: 16: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.256 INFO:tasks.ceph.mon.b.smithi143.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x561d96c365bf] 2021-03-09T09:36:06.257 INFO:tasks.ceph.mon.b.smithi143.stderr: 18: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.257 INFO:tasks.ceph.mon.b.smithi143.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7fa98f94f3f5] 2021-03-09T09:36:06.257 INFO:tasks.ceph.mon.b.smithi143.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fa98e75e6db] 2021-03-09T09:36:06.257 INFO:tasks.ceph.mon.b.smithi143.stderr: 21: clone() 2021-03-09T09:36:06.257 INFO:tasks.ceph.mon.b.smithi143.stderr:*** Caught signal (Aborted) ** 2021-03-09T09:36:06.258 INFO:tasks.ceph.mon.b.smithi143.stderr: in thread 7fa981929700 thread_name:fn_monstore 2021-03-09T09:36:06.258 INFO:tasks.ceph.mon.b.smithi143.stderr:2021-03-09T09:36:06.248+0000 7fa981929700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fa981929700 time 2021-03-09T09:36:06.242011+0000 2021-03-09T09:36:06.258 INFO:tasks.ceph.mon.b.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:06.258 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.259 INFO:tasks.ceph.mon.b.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:06.259 INFO:tasks.ceph.mon.b.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fa98f8ee3dd] 2021-03-09T09:36:06.259 INFO:tasks.ceph.mon.b.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fa98f8ee5b8] 2021-03-09T09:36:06.259 INFO:tasks.ceph.mon.b.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x561d96c45a65] 2021-03-09T09:36:06.259 INFO:tasks.ceph.mon.b.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x561d96c91a71] 2021-03-09T09:36:06.260 INFO:tasks.ceph.mon.b.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x561d96cc1295] 2021-03-09T09:36:06.260 INFO:tasks.ceph.mon.b.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x561d96cc924b] 2021-03-09T09:36:06.260 INFO:tasks.ceph.mon.b.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x561d96cc97ab] 2021-03-09T09:36:06.260 INFO:tasks.ceph.mon.b.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x561d96c38ade] 2021-03-09T09:36:06.260 INFO:tasks.ceph.mon.b.smithi143.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x561d96b8e134] 2021-03-09T09:36:06.261 INFO:tasks.ceph.mon.b.smithi143.stderr: 10: (C_MonOp::finish(int)+0x45) [0x561d96b2caa5] 2021-03-09T09:36:06.261 INFO:tasks.ceph.mon.b.smithi143.stderr: 11: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.261 INFO:tasks.ceph.mon.b.smithi143.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x561d96b55548] 2021-03-09T09:36:06.261 INFO:tasks.ceph.mon.b.smithi143.stderr: 13: (Paxos::finish_round()+0x9b) [0x561d96c307db] 2021-03-09T09:36:06.262 INFO:tasks.ceph.mon.b.smithi143.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x561d96c330c0] 2021-03-09T09:36:06.262 INFO:tasks.ceph.mon.b.smithi143.stderr: 15: (C_Committed::finish(int)+0x41) [0x561d96c36861] 2021-03-09T09:36:06.262 INFO:tasks.ceph.mon.b.smithi143.stderr: 16: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.262 INFO:tasks.ceph.mon.b.smithi143.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x561d96c365bf] 2021-03-09T09:36:06.262 INFO:tasks.ceph.mon.b.smithi143.stderr: 18: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.263 INFO:tasks.ceph.mon.b.smithi143.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7fa98f94f3f5] 2021-03-09T09:36:06.263 INFO:tasks.ceph.mon.b.smithi143.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fa98e75e6db] 2021-03-09T09:36:06.263 INFO:tasks.ceph.mon.b.smithi143.stderr: 21: clone() 2021-03-09T09:36:06.263 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.265 INFO:tasks.ceph.mon.b.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:06.265 INFO:tasks.ceph.mon.b.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fa98e769980] 2021-03-09T09:36:06.265 INFO:tasks.ceph.mon.b.smithi143.stderr: 2: gsignal() 2021-03-09T09:36:06.265 INFO:tasks.ceph.mon.b.smithi143.stderr: 3: abort() 2021-03-09T09:36:06.265 INFO:tasks.ceph.mon.b.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fa98f8ee42e] 2021-03-09T09:36:06.266 INFO:tasks.ceph.mon.b.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fa98f8ee5b8] 2021-03-09T09:36:06.266 INFO:tasks.ceph.mon.b.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x561d96c45a65] 2021-03-09T09:36:06.266 INFO:tasks.ceph.mon.b.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x561d96c91a71] 2021-03-09T09:36:06.267 INFO:tasks.ceph.mon.b.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x561d96cc1295] 2021-03-09T09:36:06.267 INFO:tasks.ceph.mon.b.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x561d96cc924b] 2021-03-09T09:36:06.267 INFO:tasks.ceph.mon.b.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x561d96cc97ab] 2021-03-09T09:36:06.267 INFO:tasks.ceph.mon.b.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x561d96c38ade] 2021-03-09T09:36:06.267 INFO:tasks.ceph.mon.b.smithi143.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x561d96b8e134] 2021-03-09T09:36:06.268 INFO:tasks.ceph.mon.b.smithi143.stderr: 13: (C_MonOp::finish(int)+0x45) [0x561d96b2caa5] 2021-03-09T09:36:06.268 INFO:tasks.ceph.mon.b.smithi143.stderr: 14: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.268 INFO:tasks.ceph.mon.b.smithi143.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x561d96b55548] 2021-03-09T09:36:06.268 INFO:tasks.ceph.mon.b.smithi143.stderr: 16: (Paxos::finish_round()+0x9b) [0x561d96c307db] 2021-03-09T09:36:06.268 INFO:tasks.ceph.mon.b.smithi143.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x561d96c330c0] 2021-03-09T09:36:06.269 INFO:tasks.ceph.mon.b.smithi143.stderr: 18: (C_Committed::finish(int)+0x41) [0x561d96c36861] 2021-03-09T09:36:06.269 INFO:tasks.ceph.mon.b.smithi143.stderr: 19: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.269 INFO:tasks.ceph.mon.b.smithi143.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x561d96c365bf] 2021-03-09T09:36:06.269 INFO:tasks.ceph.mon.b.smithi143.stderr: 21: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.270 INFO:tasks.ceph.mon.b.smithi143.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7fa98f94f3f5] 2021-03-09T09:36:06.270 INFO:tasks.ceph.mon.b.smithi143.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fa98e75e6db] 2021-03-09T09:36:06.270 INFO:tasks.ceph.mon.b.smithi143.stderr: 24: clone() 2021-03-09T09:36:06.270 INFO:tasks.ceph.mon.b.smithi143.stderr:2021-03-09T09:36:06.256+0000 7fa981929700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:36:06.270 INFO:tasks.ceph.mon.b.smithi143.stderr: in thread 7fa981929700 thread_name:fn_monstore 2021-03-09T09:36:06.271 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.271 INFO:tasks.ceph.mon.b.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:06.271 INFO:tasks.ceph.mon.b.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fa98e769980] 2021-03-09T09:36:06.271 INFO:tasks.ceph.mon.b.smithi143.stderr: 2: gsignal() 2021-03-09T09:36:06.271 INFO:tasks.ceph.mon.b.smithi143.stderr: 3: abort() 2021-03-09T09:36:06.272 INFO:tasks.ceph.mon.b.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fa98f8ee42e] 2021-03-09T09:36:06.272 INFO:tasks.ceph.mon.b.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fa98f8ee5b8] 2021-03-09T09:36:06.272 INFO:tasks.ceph.mon.b.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x561d96c45a65] 2021-03-09T09:36:06.273 INFO:tasks.ceph.mon.b.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x561d96c91a71] 2021-03-09T09:36:06.273 INFO:tasks.ceph.mon.b.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x561d96cc1295] 2021-03-09T09:36:06.273 INFO:tasks.ceph.mon.b.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x561d96cc924b] 2021-03-09T09:36:06.274 INFO:tasks.ceph.mon.b.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x561d96cc97ab] 2021-03-09T09:36:06.274 INFO:tasks.ceph.mon.b.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x561d96c38ade] 2021-03-09T09:36:06.274 INFO:tasks.ceph.mon.b.smithi143.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x561d96b8e134] 2021-03-09T09:36:06.274 INFO:tasks.ceph.mon.b.smithi143.stderr: 13: (C_MonOp::finish(int)+0x45) [0x561d96b2caa5] 2021-03-09T09:36:06.274 INFO:tasks.ceph.mon.b.smithi143.stderr: 14: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.275 INFO:tasks.ceph.mon.b.smithi143.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x561d96b55548] 2021-03-09T09:36:06.275 INFO:tasks.ceph.mon.b.smithi143.stderr: 16: (Paxos::finish_round()+0x9b) [0x561d96c307db] 2021-03-09T09:36:06.275 INFO:tasks.ceph.mon.b.smithi143.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x561d96c330c0] 2021-03-09T09:36:06.275 INFO:tasks.ceph.mon.b.smithi143.stderr: 18: (C_Committed::finish(int)+0x41) [0x561d96c36861] 2021-03-09T09:36:06.276 INFO:tasks.ceph.mon.b.smithi143.stderr: 19: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.276 INFO:tasks.ceph.mon.b.smithi143.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x561d96c365bf] 2021-03-09T09:36:06.276 INFO:tasks.ceph.mon.b.smithi143.stderr: 21: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.276 INFO:tasks.ceph.mon.b.smithi143.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7fa98f94f3f5] 2021-03-09T09:36:06.276 INFO:tasks.ceph.mon.b.smithi143.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fa98e75e6db] 2021-03-09T09:36:06.277 INFO:tasks.ceph.mon.b.smithi143.stderr: 24: clone() 2021-03-09T09:36:06.277 INFO:tasks.ceph.mon.b.smithi143.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:36:06.277 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.278 INFO:tasks.ceph.mon.b.smithi143.stderr: -2315> 2021-03-09T09:36:01.662+0000 7fa985931700 -1 mon.b@2(electing) e1 get_health_metrics reporting 11 slow ops, oldest is mon_command({"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1} v 0) 2021-03-09T09:36:06.278 INFO:tasks.ceph.mon.b.smithi143.stderr: -1> 2021-03-09T09:36:06.248+0000 7fa981929700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fa981929700 time 2021-03-09T09:36:06.242011+0000 2021-03-09T09:36:06.278 INFO:tasks.ceph.mon.b.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:06.278 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.279 INFO:tasks.ceph.mon.b.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:06.279 INFO:tasks.ceph.mon.b.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fa98f8ee3dd] 2021-03-09T09:36:06.279 INFO:tasks.ceph.mon.b.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fa98f8ee5b8] 2021-03-09T09:36:06.279 INFO:tasks.ceph.mon.b.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x561d96c45a65] 2021-03-09T09:36:06.279 INFO:tasks.ceph.mon.b.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x561d96c91a71] 2021-03-09T09:36:06.280 INFO:tasks.ceph.mon.b.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x561d96cc1295] 2021-03-09T09:36:06.280 INFO:tasks.ceph.mon.b.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x561d96cc924b] 2021-03-09T09:36:06.280 INFO:tasks.ceph.mon.b.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x561d96cc97ab] 2021-03-09T09:36:06.280 INFO:tasks.ceph.mon.b.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x561d96c38ade] 2021-03-09T09:36:06.281 INFO:tasks.ceph.mon.b.smithi143.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x561d96b8e134] 2021-03-09T09:36:06.281 INFO:tasks.ceph.mon.b.smithi143.stderr: 10: (C_MonOp::finish(int)+0x45) [0x561d96b2caa5] 2021-03-09T09:36:06.281 INFO:tasks.ceph.mon.b.smithi143.stderr: 11: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.281 INFO:tasks.ceph.mon.b.smithi143.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x561d96b55548] 2021-03-09T09:36:06.281 INFO:tasks.ceph.mon.b.smithi143.stderr: 13: (Paxos::finish_round()+0x9b) [0x561d96c307db] 2021-03-09T09:36:06.282 INFO:tasks.ceph.mon.b.smithi143.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x561d96c330c0] 2021-03-09T09:36:06.282 INFO:tasks.ceph.mon.b.smithi143.stderr: 15: (C_Committed::finish(int)+0x41) [0x561d96c36861] 2021-03-09T09:36:06.282 INFO:tasks.ceph.mon.b.smithi143.stderr: 16: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.282 INFO:tasks.ceph.mon.b.smithi143.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x561d96c365bf] 2021-03-09T09:36:06.282 INFO:tasks.ceph.mon.b.smithi143.stderr: 18: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.283 INFO:tasks.ceph.mon.b.smithi143.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7fa98f94f3f5] 2021-03-09T09:36:06.283 INFO:tasks.ceph.mon.b.smithi143.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fa98e75e6db] 2021-03-09T09:36:06.283 INFO:tasks.ceph.mon.b.smithi143.stderr: 21: clone() 2021-03-09T09:36:06.283 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.284 INFO:tasks.ceph.mon.b.smithi143.stderr: 0> 2021-03-09T09:36:06.256+0000 7fa981929700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:36:06.284 INFO:tasks.ceph.mon.b.smithi143.stderr: in thread 7fa981929700 thread_name:fn_monstore 2021-03-09T09:36:06.284 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.285 INFO:tasks.ceph.mon.b.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:06.285 INFO:tasks.ceph.mon.b.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fa98e769980] 2021-03-09T09:36:06.285 INFO:tasks.ceph.mon.b.smithi143.stderr: 2: gsignal() 2021-03-09T09:36:06.285 INFO:tasks.ceph.mon.b.smithi143.stderr: 3: abort() 2021-03-09T09:36:06.285 INFO:tasks.ceph.mon.b.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fa98f8ee42e] 2021-03-09T09:36:06.286 INFO:tasks.ceph.mon.b.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fa98f8ee5b8] 2021-03-09T09:36:06.286 INFO:tasks.ceph.mon.b.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x561d96c45a65] 2021-03-09T09:36:06.286 INFO:tasks.ceph.mon.b.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x561d96c91a71] 2021-03-09T09:36:06.286 INFO:tasks.ceph.mon.b.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x561d96cc1295] 2021-03-09T09:36:06.287 INFO:tasks.ceph.mon.b.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x561d96cc924b] 2021-03-09T09:36:06.287 INFO:tasks.ceph.mon.b.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x561d96cc97ab] 2021-03-09T09:36:06.287 INFO:tasks.ceph.mon.b.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x561d96c38ade] 2021-03-09T09:36:06.287 INFO:tasks.ceph.mon.b.smithi143.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x561d96b8e134] 2021-03-09T09:36:06.287 INFO:tasks.ceph.mon.b.smithi143.stderr: 13: (C_MonOp::finish(int)+0x45) [0x561d96b2caa5] 2021-03-09T09:36:06.288 INFO:tasks.ceph.mon.b.smithi143.stderr: 14: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.288 INFO:tasks.ceph.mon.b.smithi143.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x561d96b55548] 2021-03-09T09:36:06.288 INFO:tasks.ceph.mon.b.smithi143.stderr: 16: (Paxos::finish_round()+0x9b) [0x561d96c307db] 2021-03-09T09:36:06.288 INFO:tasks.ceph.mon.b.smithi143.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x561d96c330c0] 2021-03-09T09:36:06.289 INFO:tasks.ceph.mon.b.smithi143.stderr: 18: (C_Committed::finish(int)+0x41) [0x561d96c36861] 2021-03-09T09:36:06.289 INFO:tasks.ceph.mon.b.smithi143.stderr: 19: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.289 INFO:tasks.ceph.mon.b.smithi143.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x561d96c365bf] 2021-03-09T09:36:06.289 INFO:tasks.ceph.mon.b.smithi143.stderr: 21: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.289 INFO:tasks.ceph.mon.b.smithi143.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7fa98f94f3f5] 2021-03-09T09:36:06.290 INFO:tasks.ceph.mon.b.smithi143.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fa98e75e6db] 2021-03-09T09:36:06.290 INFO:tasks.ceph.mon.b.smithi143.stderr: 24: clone() 2021-03-09T09:36:06.290 INFO:tasks.ceph.mon.b.smithi143.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:36:06.290 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.291 INFO:tasks.ceph.mon.b.smithi143.stderr: -2315> 2021-03-09T09:36:01.662+0000 7fa985931700 -1 mon.b@2(electing) e1 get_health_metrics reporting 11 slow ops, oldest is mon_command({"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1} v 0) 2021-03-09T09:36:06.291 INFO:tasks.ceph.mon.b.smithi143.stderr: -1> 2021-03-09T09:36:06.248+0000 7fa981929700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7fa981929700 time 2021-03-09T09:36:06.242011+0000 2021-03-09T09:36:06.291 INFO:tasks.ceph.mon.b.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:06.292 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.292 INFO:tasks.ceph.mon.b.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:06.292 INFO:tasks.ceph.mon.b.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fa98f8ee3dd] 2021-03-09T09:36:06.292 INFO:tasks.ceph.mon.b.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fa98f8ee5b8] 2021-03-09T09:36:06.293 INFO:tasks.ceph.mon.b.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x561d96c45a65] 2021-03-09T09:36:06.293 INFO:tasks.ceph.mon.b.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x561d96c91a71] 2021-03-09T09:36:06.293 INFO:tasks.ceph.mon.b.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x561d96cc1295] 2021-03-09T09:36:06.293 INFO:tasks.ceph.mon.b.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x561d96cc924b] 2021-03-09T09:36:06.293 INFO:tasks.ceph.mon.b.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x561d96cc97ab] 2021-03-09T09:36:06.294 INFO:tasks.ceph.mon.b.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x561d96c38ade] 2021-03-09T09:36:06.294 INFO:tasks.ceph.mon.b.smithi143.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x561d96b8e134] 2021-03-09T09:36:06.294 INFO:tasks.ceph.mon.b.smithi143.stderr: 10: (C_MonOp::finish(int)+0x45) [0x561d96b2caa5] 2021-03-09T09:36:06.294 INFO:tasks.ceph.mon.b.smithi143.stderr: 11: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.294 INFO:tasks.ceph.mon.b.smithi143.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x561d96b55548] 2021-03-09T09:36:06.295 INFO:tasks.ceph.mon.b.smithi143.stderr: 13: (Paxos::finish_round()+0x9b) [0x561d96c307db] 2021-03-09T09:36:06.295 INFO:tasks.ceph.mon.b.smithi143.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x561d96c330c0] 2021-03-09T09:36:06.295 INFO:tasks.ceph.mon.b.smithi143.stderr: 15: (C_Committed::finish(int)+0x41) [0x561d96c36861] 2021-03-09T09:36:06.295 INFO:tasks.ceph.mon.b.smithi143.stderr: 16: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.295 INFO:tasks.ceph.mon.b.smithi143.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x561d96c365bf] 2021-03-09T09:36:06.296 INFO:tasks.ceph.mon.b.smithi143.stderr: 18: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.296 INFO:tasks.ceph.mon.b.smithi143.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7fa98f94f3f5] 2021-03-09T09:36:06.296 INFO:tasks.ceph.mon.b.smithi143.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fa98e75e6db] 2021-03-09T09:36:06.296 INFO:tasks.ceph.mon.b.smithi143.stderr: 21: clone() 2021-03-09T09:36:06.297 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.297 INFO:tasks.ceph.mon.b.smithi143.stderr: 0> 2021-03-09T09:36:06.256+0000 7fa981929700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:36:06.297 INFO:tasks.ceph.mon.b.smithi143.stderr: in thread 7fa981929700 thread_name:fn_monstore 2021-03-09T09:36:06.297 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.297 INFO:tasks.ceph.mon.b.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:06.298 INFO:tasks.ceph.mon.b.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fa98e769980] 2021-03-09T09:36:06.298 INFO:tasks.ceph.mon.b.smithi143.stderr: 2: gsignal() 2021-03-09T09:36:06.298 INFO:tasks.ceph.mon.b.smithi143.stderr: 3: abort() 2021-03-09T09:36:06.298 INFO:tasks.ceph.mon.b.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fa98f8ee42e] 2021-03-09T09:36:06.298 INFO:tasks.ceph.mon.b.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fa98f8ee5b8] 2021-03-09T09:36:06.299 INFO:tasks.ceph.mon.b.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x561d96c45a65] 2021-03-09T09:36:06.299 INFO:tasks.ceph.mon.b.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x561d96c91a71] 2021-03-09T09:36:06.299 INFO:tasks.ceph.mon.b.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x561d96cc1295] 2021-03-09T09:36:06.299 INFO:tasks.ceph.mon.b.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x561d96cc924b] 2021-03-09T09:36:06.300 INFO:tasks.ceph.mon.b.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x561d96cc97ab] 2021-03-09T09:36:06.300 INFO:tasks.ceph.mon.b.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x561d96c38ade] 2021-03-09T09:36:06.300 INFO:tasks.ceph.mon.b.smithi143.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x561d96b8e134] 2021-03-09T09:36:06.300 INFO:tasks.ceph.mon.b.smithi143.stderr: 13: (C_MonOp::finish(int)+0x45) [0x561d96b2caa5] 2021-03-09T09:36:06.300 INFO:tasks.ceph.mon.b.smithi143.stderr: 14: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.301 INFO:tasks.ceph.mon.b.smithi143.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x561d96b55548] 2021-03-09T09:36:06.301 INFO:tasks.ceph.mon.b.smithi143.stderr: 16: (Paxos::finish_round()+0x9b) [0x561d96c307db] 2021-03-09T09:36:06.301 INFO:tasks.ceph.mon.b.smithi143.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x561d96c330c0] 2021-03-09T09:36:06.301 INFO:tasks.ceph.mon.b.smithi143.stderr: 18: (C_Committed::finish(int)+0x41) [0x561d96c36861] 2021-03-09T09:36:06.301 INFO:tasks.ceph.mon.b.smithi143.stderr: 19: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.302 INFO:tasks.ceph.mon.b.smithi143.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x561d96c365bf] 2021-03-09T09:36:06.302 INFO:tasks.ceph.mon.b.smithi143.stderr: 21: (Context::complete(int)+0x9) [0x561d96b2a059] 2021-03-09T09:36:06.302 INFO:tasks.ceph.mon.b.smithi143.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7fa98f94f3f5] 2021-03-09T09:36:06.302 INFO:tasks.ceph.mon.b.smithi143.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fa98e75e6db] 2021-03-09T09:36:06.303 INFO:tasks.ceph.mon.b.smithi143.stderr: 24: clone() 2021-03-09T09:36:06.303 INFO:tasks.ceph.mon.b.smithi143.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:36:06.303 INFO:tasks.ceph.mon.b.smithi143.stderr: 2021-03-09T09:36:06.304 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:06.292+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:06.304 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:06.292+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:06.667 INFO:tasks.ceph.mon.c.smithi143.stderr:2021-03-09T09:36:06.664+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:06.678 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:06.676+0000 7fd6e9553700 -1 mon.e@8(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:06.703 INFO:tasks.ceph.mon.g.smithi173.stderr:2021-03-09T09:36:06.701+0000 7f2a6dca2700 -1 mon.g@3(peon) e1 get_health_metrics reporting 4 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:06.732 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:06.733+0000 7fcabb99b700 -1 mon.h@5(peon) e1 get_health_metrics reporting 4 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:07.022 INFO:tasks.ceph.mon.b.smithi143.stderr:daemon-helper: command crashed with signal 6 2021-03-09T09:36:07.294 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:07.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:07.294 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:07.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:08.294 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:08.295+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:08.295 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:08.295+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:09.295 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:09.295+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:09.295 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:09.295+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:10.295 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:10.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:10.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:10.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:11.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:11.294+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:11.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:11.294+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:11.669 INFO:tasks.ceph.mon.c.smithi143.stderr:2021-03-09T09:36:11.666+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:11.680 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:11.678+0000 7fd6e9553700 -1 mon.e@8(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:11.682 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:11.678+0000 7f445d90b700 -1 mon.d@6(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:11.704 INFO:tasks.ceph.mon.g.smithi173.stderr:2021-03-09T09:36:11.704+0000 7f2a6dca2700 -1 mon.g@3(peon) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:11.729 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:11.732+0000 7fbab2ccd700 -1 mon.i@7(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:11.734 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:11.735+0000 7fcabb99b700 -1 mon.h@5(peon) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:12.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:12.293+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:12.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:12.293+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:13.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:13.293+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:13.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:13.293+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:14.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:14.293+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:14.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:14.293+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:15.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:15.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:15.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:15.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:15.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:15.296+0000 7ff35e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:16.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:16.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:16.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:16.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:16.671 INFO:tasks.ceph.mon.c.smithi143.stderr:2021-03-09T09:36:16.668+0000 7f4e89a2e700 -1 mon.c@4(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:16.682 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:16.676+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:16.684 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:16.676+0000 7f445d90b700 -1 mon.d@6(electing) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:16.707 INFO:tasks.ceph.mon.g.smithi173.stderr:2021-03-09T09:36:16.701+0000 7f2a6dca2700 -1 mon.g@3(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:16.732 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:16.729+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:16.736 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:16.733+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:17.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:17.292+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:17.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:17.292+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:18.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:18.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:18.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:18.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:19.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:19.292+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:19.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:19.292+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:20.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:20.292+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:20.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:20.292+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:21.263 INFO:tasks.ceph.mon.g.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f2a69c9a700 time 2021-03-09T09:36:21.262288+0000 2021-03-09T09:36:21.264 INFO:tasks.ceph.mon.g.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:21.273 INFO:tasks.ceph.mon.g.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:21.274 INFO:tasks.ceph.mon.g.smithi173.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f2a77c5f3dd] 2021-03-09T09:36:21.274 INFO:tasks.ceph.mon.g.smithi173.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2a77c5f5b8] 2021-03-09T09:36:21.275 INFO:tasks.ceph.mon.g.smithi173.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b6f5726a65] 2021-03-09T09:36:21.275 INFO:tasks.ceph.mon.g.smithi173.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b6f5772a71] 2021-03-09T09:36:21.276 INFO:tasks.ceph.mon.g.smithi173.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b6f57a2295] 2021-03-09T09:36:21.276 INFO:tasks.ceph.mon.g.smithi173.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b6f57aa24b] 2021-03-09T09:36:21.277 INFO:tasks.ceph.mon.g.smithi173.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b6f57aa7ab] 2021-03-09T09:36:21.277 INFO:tasks.ceph.mon.g.smithi173.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b6f5719ade] 2021-03-09T09:36:21.278 INFO:tasks.ceph.mon.g.smithi173.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55b6f566f134] 2021-03-09T09:36:21.278 INFO:tasks.ceph.mon.g.smithi173.stderr: 10: (C_MonOp::finish(int)+0x45) [0x55b6f560daa5] 2021-03-09T09:36:21.279 INFO:tasks.ceph.mon.g.smithi173.stderr: 11: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.279 INFO:tasks.ceph.mon.g.smithi173.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55b6f5636548] 2021-03-09T09:36:21.280 INFO:tasks.ceph.mon.g.smithi173.stderr: 13: (Paxos::finish_round()+0x9b) [0x55b6f57117db] 2021-03-09T09:36:21.280 INFO:tasks.ceph.mon.g.smithi173.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x55b6f57140c0] 2021-03-09T09:36:21.281 INFO:tasks.ceph.mon.g.smithi173.stderr: 15: (C_Committed::finish(int)+0x41) [0x55b6f5717861] 2021-03-09T09:36:21.281 INFO:tasks.ceph.mon.g.smithi173.stderr: 16: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.282 INFO:tasks.ceph.mon.g.smithi173.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55b6f57175bf] 2021-03-09T09:36:21.282 INFO:tasks.ceph.mon.g.smithi173.stderr: 18: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.283 INFO:tasks.ceph.mon.g.smithi173.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7f2a77cc03f5] 2021-03-09T09:36:21.283 INFO:tasks.ceph.mon.g.smithi173.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2a76acf6db] 2021-03-09T09:36:21.283 INFO:tasks.ceph.mon.g.smithi173.stderr: 21: clone() 2021-03-09T09:36:21.284 INFO:tasks.ceph.mon.g.smithi173.stderr:*** Caught signal (Aborted) ** 2021-03-09T09:36:21.284 INFO:tasks.ceph.mon.g.smithi173.stderr: in thread 7f2a69c9a700 thread_name:fn_monstore 2021-03-09T09:36:21.284 INFO:tasks.ceph.mon.g.smithi173.stderr:2021-03-09T09:36:21.268+0000 7f2a69c9a700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f2a69c9a700 time 2021-03-09T09:36:21.262288+0000 2021-03-09T09:36:21.285 INFO:tasks.ceph.mon.g.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:21.285 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.285 INFO:tasks.ceph.mon.g.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:21.286 INFO:tasks.ceph.mon.g.smithi173.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f2a77c5f3dd] 2021-03-09T09:36:21.286 INFO:tasks.ceph.mon.g.smithi173.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2a77c5f5b8] 2021-03-09T09:36:21.287 INFO:tasks.ceph.mon.g.smithi173.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b6f5726a65] 2021-03-09T09:36:21.288 INFO:tasks.ceph.mon.g.smithi173.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b6f5772a71] 2021-03-09T09:36:21.288 INFO:tasks.ceph.mon.g.smithi173.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b6f57a2295] 2021-03-09T09:36:21.288 INFO:tasks.ceph.mon.g.smithi173.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b6f57aa24b] 2021-03-09T09:36:21.289 INFO:tasks.ceph.mon.g.smithi173.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b6f57aa7ab] 2021-03-09T09:36:21.289 INFO:tasks.ceph.mon.g.smithi173.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b6f5719ade] 2021-03-09T09:36:21.289 INFO:tasks.ceph.mon.g.smithi173.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55b6f566f134] 2021-03-09T09:36:21.290 INFO:tasks.ceph.mon.g.smithi173.stderr: 10: (C_MonOp::finish(int)+0x45) [0x55b6f560daa5] 2021-03-09T09:36:21.290 INFO:tasks.ceph.mon.g.smithi173.stderr: 11: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.290 INFO:tasks.ceph.mon.g.smithi173.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55b6f5636548] 2021-03-09T09:36:21.290 INFO:tasks.ceph.mon.g.smithi173.stderr: 13: (Paxos::finish_round()+0x9b) [0x55b6f57117db] 2021-03-09T09:36:21.291 INFO:tasks.ceph.mon.g.smithi173.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x55b6f57140c0] 2021-03-09T09:36:21.291 INFO:tasks.ceph.mon.g.smithi173.stderr: 15: (C_Committed::finish(int)+0x41) [0x55b6f5717861] 2021-03-09T09:36:21.291 INFO:tasks.ceph.mon.g.smithi173.stderr: 16: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.291 INFO:tasks.ceph.mon.g.smithi173.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55b6f57175bf] 2021-03-09T09:36:21.291 INFO:tasks.ceph.mon.g.smithi173.stderr: 18: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.292 INFO:tasks.ceph.mon.g.smithi173.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7f2a77cc03f5] 2021-03-09T09:36:21.292 INFO:tasks.ceph.mon.g.smithi173.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2a76acf6db] 2021-03-09T09:36:21.292 INFO:tasks.ceph.mon.g.smithi173.stderr: 21: clone() 2021-03-09T09:36:21.292 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.293 INFO:tasks.ceph.mon.g.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:21.294 INFO:tasks.ceph.mon.g.smithi173.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f2a76ada980] 2021-03-09T09:36:21.294 INFO:tasks.ceph.mon.g.smithi173.stderr: 2: gsignal() 2021-03-09T09:36:21.295 INFO:tasks.ceph.mon.g.smithi173.stderr: 3: abort() 2021-03-09T09:36:21.295 INFO:tasks.ceph.mon.g.smithi173.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f2a77c5f42e] 2021-03-09T09:36:21.295 INFO:tasks.ceph.mon.g.smithi173.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2a77c5f5b8] 2021-03-09T09:36:21.295 INFO:tasks.ceph.mon.g.smithi173.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b6f5726a65] 2021-03-09T09:36:21.296 INFO:tasks.ceph.mon.g.smithi173.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b6f5772a71] 2021-03-09T09:36:21.296 INFO:tasks.ceph.mon.g.smithi173.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b6f57a2295] 2021-03-09T09:36:21.297 INFO:tasks.ceph.mon.g.smithi173.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b6f57aa24b] 2021-03-09T09:36:21.297 INFO:tasks.ceph.mon.g.smithi173.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b6f57aa7ab] 2021-03-09T09:36:21.297 INFO:tasks.ceph.mon.g.smithi173.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b6f5719ade] 2021-03-09T09:36:21.298 INFO:tasks.ceph.mon.g.smithi173.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55b6f566f134] 2021-03-09T09:36:21.298 INFO:tasks.ceph.mon.g.smithi173.stderr: 13: (C_MonOp::finish(int)+0x45) [0x55b6f560daa5] 2021-03-09T09:36:21.299 INFO:tasks.ceph.mon.g.smithi173.stderr: 14: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.299 INFO:tasks.ceph.mon.g.smithi173.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55b6f5636548] 2021-03-09T09:36:21.300 INFO:tasks.ceph.mon.g.smithi173.stderr: 16: (Paxos::finish_round()+0x9b) [0x55b6f57117db] 2021-03-09T09:36:21.300 INFO:tasks.ceph.mon.g.smithi173.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x55b6f57140c0] 2021-03-09T09:36:21.300 INFO:tasks.ceph.mon.g.smithi173.stderr: 18: (C_Committed::finish(int)+0x41) [0x55b6f5717861] 2021-03-09T09:36:21.301 INFO:tasks.ceph.mon.g.smithi173.stderr: 19: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.301 INFO:tasks.ceph.mon.g.smithi173.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55b6f57175bf] 2021-03-09T09:36:21.302 INFO:tasks.ceph.mon.g.smithi173.stderr: 21: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.302 INFO:tasks.ceph.mon.g.smithi173.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7f2a77cc03f5] 2021-03-09T09:36:21.302 INFO:tasks.ceph.mon.g.smithi173.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2a76acf6db] 2021-03-09T09:36:21.303 INFO:tasks.ceph.mon.g.smithi173.stderr: 24: clone() 2021-03-09T09:36:21.303 INFO:tasks.ceph.mon.g.smithi173.stderr:2021-03-09T09:36:21.276+0000 7f2a69c9a700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:36:21.303 INFO:tasks.ceph.mon.g.smithi173.stderr: in thread 7f2a69c9a700 thread_name:fn_monstore 2021-03-09T09:36:21.304 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.304 INFO:tasks.ceph.mon.g.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:21.304 INFO:tasks.ceph.mon.g.smithi173.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f2a76ada980] 2021-03-09T09:36:21.305 INFO:tasks.ceph.mon.g.smithi173.stderr: 2: gsignal() 2021-03-09T09:36:21.305 INFO:tasks.ceph.mon.g.smithi173.stderr: 3: abort() 2021-03-09T09:36:21.306 INFO:tasks.ceph.mon.g.smithi173.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f2a77c5f42e] 2021-03-09T09:36:21.306 INFO:tasks.ceph.mon.g.smithi173.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2a77c5f5b8] 2021-03-09T09:36:21.306 INFO:tasks.ceph.mon.g.smithi173.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b6f5726a65] 2021-03-09T09:36:21.307 INFO:tasks.ceph.mon.g.smithi173.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b6f5772a71] 2021-03-09T09:36:21.307 INFO:tasks.ceph.mon.g.smithi173.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b6f57a2295] 2021-03-09T09:36:21.307 INFO:tasks.ceph.mon.g.smithi173.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b6f57aa24b] 2021-03-09T09:36:21.308 INFO:tasks.ceph.mon.g.smithi173.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b6f57aa7ab] 2021-03-09T09:36:21.308 INFO:tasks.ceph.mon.g.smithi173.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b6f5719ade] 2021-03-09T09:36:21.308 INFO:tasks.ceph.mon.g.smithi173.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55b6f566f134] 2021-03-09T09:36:21.309 INFO:tasks.ceph.mon.g.smithi173.stderr: 13: (C_MonOp::finish(int)+0x45) [0x55b6f560daa5] 2021-03-09T09:36:21.309 INFO:tasks.ceph.mon.g.smithi173.stderr: 14: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.309 INFO:tasks.ceph.mon.g.smithi173.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55b6f5636548] 2021-03-09T09:36:21.310 INFO:tasks.ceph.mon.g.smithi173.stderr: 16: (Paxos::finish_round()+0x9b) [0x55b6f57117db] 2021-03-09T09:36:21.310 INFO:tasks.ceph.mon.g.smithi173.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x55b6f57140c0] 2021-03-09T09:36:21.311 INFO:tasks.ceph.mon.g.smithi173.stderr: 18: (C_Committed::finish(int)+0x41) [0x55b6f5717861] 2021-03-09T09:36:21.311 INFO:tasks.ceph.mon.g.smithi173.stderr: 19: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.311 INFO:tasks.ceph.mon.g.smithi173.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55b6f57175bf] 2021-03-09T09:36:21.311 INFO:tasks.ceph.mon.g.smithi173.stderr: 21: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.312 INFO:tasks.ceph.mon.g.smithi173.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7f2a77cc03f5] 2021-03-09T09:36:21.312 INFO:tasks.ceph.mon.g.smithi173.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2a76acf6db] 2021-03-09T09:36:21.312 INFO:tasks.ceph.mon.g.smithi173.stderr: 24: clone() 2021-03-09T09:36:21.312 INFO:tasks.ceph.mon.g.smithi173.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:36:21.313 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.314 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:21.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:21.314 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:21.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:21.316 INFO:tasks.ceph.mon.g.smithi173.stderr: -4565> 2021-03-09T09:36:01.705+0000 7f2a6dca2700 -1 mon.g@3(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:21.316 INFO:tasks.ceph.mon.g.smithi173.stderr: -3523> 2021-03-09T09:36:06.701+0000 7f2a6dca2700 -1 mon.g@3(peon) e1 get_health_metrics reporting 4 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:21.316 INFO:tasks.ceph.mon.g.smithi173.stderr: -2847> 2021-03-09T09:36:11.704+0000 7f2a6dca2700 -1 mon.g@3(peon) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:21.316 INFO:tasks.ceph.mon.g.smithi173.stderr: -1849> 2021-03-09T09:36:16.701+0000 7f2a6dca2700 -1 mon.g@3(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:21.317 INFO:tasks.ceph.mon.g.smithi173.stderr: -1> 2021-03-09T09:36:21.268+0000 7f2a69c9a700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f2a69c9a700 time 2021-03-09T09:36:21.262288+0000 2021-03-09T09:36:21.317 INFO:tasks.ceph.mon.g.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:21.317 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.317 INFO:tasks.ceph.mon.g.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:21.317 INFO:tasks.ceph.mon.g.smithi173.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f2a77c5f3dd] 2021-03-09T09:36:21.318 INFO:tasks.ceph.mon.g.smithi173.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2a77c5f5b8] 2021-03-09T09:36:21.318 INFO:tasks.ceph.mon.g.smithi173.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b6f5726a65] 2021-03-09T09:36:21.318 INFO:tasks.ceph.mon.g.smithi173.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b6f5772a71] 2021-03-09T09:36:21.318 INFO:tasks.ceph.mon.g.smithi173.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b6f57a2295] 2021-03-09T09:36:21.319 INFO:tasks.ceph.mon.g.smithi173.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b6f57aa24b] 2021-03-09T09:36:21.319 INFO:tasks.ceph.mon.g.smithi173.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b6f57aa7ab] 2021-03-09T09:36:21.319 INFO:tasks.ceph.mon.g.smithi173.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b6f5719ade] 2021-03-09T09:36:21.319 INFO:tasks.ceph.mon.g.smithi173.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55b6f566f134] 2021-03-09T09:36:21.319 INFO:tasks.ceph.mon.g.smithi173.stderr: 10: (C_MonOp::finish(int)+0x45) [0x55b6f560daa5] 2021-03-09T09:36:21.320 INFO:tasks.ceph.mon.g.smithi173.stderr: 11: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.320 INFO:tasks.ceph.mon.g.smithi173.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55b6f5636548] 2021-03-09T09:36:21.320 INFO:tasks.ceph.mon.g.smithi173.stderr: 13: (Paxos::finish_round()+0x9b) [0x55b6f57117db] 2021-03-09T09:36:21.320 INFO:tasks.ceph.mon.g.smithi173.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x55b6f57140c0] 2021-03-09T09:36:21.320 INFO:tasks.ceph.mon.g.smithi173.stderr: 15: (C_Committed::finish(int)+0x41) [0x55b6f5717861] 2021-03-09T09:36:21.321 INFO:tasks.ceph.mon.g.smithi173.stderr: 16: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.321 INFO:tasks.ceph.mon.g.smithi173.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55b6f57175bf] 2021-03-09T09:36:21.321 INFO:tasks.ceph.mon.g.smithi173.stderr: 18: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.322 INFO:tasks.ceph.mon.g.smithi173.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7f2a77cc03f5] 2021-03-09T09:36:21.322 INFO:tasks.ceph.mon.g.smithi173.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2a76acf6db] 2021-03-09T09:36:21.322 INFO:tasks.ceph.mon.g.smithi173.stderr: 21: clone() 2021-03-09T09:36:21.322 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.323 INFO:tasks.ceph.mon.g.smithi173.stderr: 0> 2021-03-09T09:36:21.276+0000 7f2a69c9a700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:36:21.323 INFO:tasks.ceph.mon.g.smithi173.stderr: in thread 7f2a69c9a700 thread_name:fn_monstore 2021-03-09T09:36:21.323 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.323 INFO:tasks.ceph.mon.g.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:21.323 INFO:tasks.ceph.mon.g.smithi173.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f2a76ada980] 2021-03-09T09:36:21.324 INFO:tasks.ceph.mon.g.smithi173.stderr: 2: gsignal() 2021-03-09T09:36:21.324 INFO:tasks.ceph.mon.g.smithi173.stderr: 3: abort() 2021-03-09T09:36:21.324 INFO:tasks.ceph.mon.g.smithi173.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f2a77c5f42e] 2021-03-09T09:36:21.324 INFO:tasks.ceph.mon.g.smithi173.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2a77c5f5b8] 2021-03-09T09:36:21.324 INFO:tasks.ceph.mon.g.smithi173.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b6f5726a65] 2021-03-09T09:36:21.325 INFO:tasks.ceph.mon.g.smithi173.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b6f5772a71] 2021-03-09T09:36:21.325 INFO:tasks.ceph.mon.g.smithi173.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b6f57a2295] 2021-03-09T09:36:21.325 INFO:tasks.ceph.mon.g.smithi173.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b6f57aa24b] 2021-03-09T09:36:21.325 INFO:tasks.ceph.mon.g.smithi173.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b6f57aa7ab] 2021-03-09T09:36:21.326 INFO:tasks.ceph.mon.g.smithi173.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b6f5719ade] 2021-03-09T09:36:21.326 INFO:tasks.ceph.mon.g.smithi173.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55b6f566f134] 2021-03-09T09:36:21.326 INFO:tasks.ceph.mon.g.smithi173.stderr: 13: (C_MonOp::finish(int)+0x45) [0x55b6f560daa5] 2021-03-09T09:36:21.326 INFO:tasks.ceph.mon.g.smithi173.stderr: 14: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.326 INFO:tasks.ceph.mon.g.smithi173.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55b6f5636548] 2021-03-09T09:36:21.327 INFO:tasks.ceph.mon.g.smithi173.stderr: 16: (Paxos::finish_round()+0x9b) [0x55b6f57117db] 2021-03-09T09:36:21.327 INFO:tasks.ceph.mon.g.smithi173.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x55b6f57140c0] 2021-03-09T09:36:21.327 INFO:tasks.ceph.mon.g.smithi173.stderr: 18: (C_Committed::finish(int)+0x41) [0x55b6f5717861] 2021-03-09T09:36:21.328 INFO:tasks.ceph.mon.g.smithi173.stderr: 19: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.328 INFO:tasks.ceph.mon.g.smithi173.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55b6f57175bf] 2021-03-09T09:36:21.328 INFO:tasks.ceph.mon.g.smithi173.stderr: 21: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.329 INFO:tasks.ceph.mon.g.smithi173.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7f2a77cc03f5] 2021-03-09T09:36:21.329 INFO:tasks.ceph.mon.g.smithi173.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2a76acf6db] 2021-03-09T09:36:21.330 INFO:tasks.ceph.mon.g.smithi173.stderr: 24: clone() 2021-03-09T09:36:21.330 INFO:tasks.ceph.mon.g.smithi173.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:36:21.331 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.331 INFO:tasks.ceph.mon.g.smithi173.stderr: -4565> 2021-03-09T09:36:01.705+0000 7f2a6dca2700 -1 mon.g@3(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:21.332 INFO:tasks.ceph.mon.g.smithi173.stderr: -3523> 2021-03-09T09:36:06.701+0000 7f2a6dca2700 -1 mon.g@3(peon) e1 get_health_metrics reporting 4 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:21.332 INFO:tasks.ceph.mon.g.smithi173.stderr: -2847> 2021-03-09T09:36:11.704+0000 7f2a6dca2700 -1 mon.g@3(peon) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:21.332 INFO:tasks.ceph.mon.g.smithi173.stderr: -1849> 2021-03-09T09:36:16.701+0000 7f2a6dca2700 -1 mon.g@3(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.599260+0000) 2021-03-09T09:36:21.333 INFO:tasks.ceph.mon.g.smithi173.stderr: -1> 2021-03-09T09:36:21.268+0000 7f2a69c9a700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f2a69c9a700 time 2021-03-09T09:36:21.262288+0000 2021-03-09T09:36:21.333 INFO:tasks.ceph.mon.g.smithi173.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:21.334 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.334 INFO:tasks.ceph.mon.g.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:21.335 INFO:tasks.ceph.mon.g.smithi173.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f2a77c5f3dd] 2021-03-09T09:36:21.335 INFO:tasks.ceph.mon.g.smithi173.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2a77c5f5b8] 2021-03-09T09:36:21.335 INFO:tasks.ceph.mon.g.smithi173.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b6f5726a65] 2021-03-09T09:36:21.336 INFO:tasks.ceph.mon.g.smithi173.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b6f5772a71] 2021-03-09T09:36:21.336 INFO:tasks.ceph.mon.g.smithi173.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b6f57a2295] 2021-03-09T09:36:21.337 INFO:tasks.ceph.mon.g.smithi173.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b6f57aa24b] 2021-03-09T09:36:21.337 INFO:tasks.ceph.mon.g.smithi173.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b6f57aa7ab] 2021-03-09T09:36:21.338 INFO:tasks.ceph.mon.g.smithi173.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b6f5719ade] 2021-03-09T09:36:21.338 INFO:tasks.ceph.mon.g.smithi173.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55b6f566f134] 2021-03-09T09:36:21.338 INFO:tasks.ceph.mon.g.smithi173.stderr: 10: (C_MonOp::finish(int)+0x45) [0x55b6f560daa5] 2021-03-09T09:36:21.339 INFO:tasks.ceph.mon.g.smithi173.stderr: 11: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.339 INFO:tasks.ceph.mon.g.smithi173.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55b6f5636548] 2021-03-09T09:36:21.339 INFO:tasks.ceph.mon.g.smithi173.stderr: 13: (Paxos::finish_round()+0x9b) [0x55b6f57117db] 2021-03-09T09:36:21.339 INFO:tasks.ceph.mon.g.smithi173.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x55b6f57140c0] 2021-03-09T09:36:21.340 INFO:tasks.ceph.mon.g.smithi173.stderr: 15: (C_Committed::finish(int)+0x41) [0x55b6f5717861] 2021-03-09T09:36:21.340 INFO:tasks.ceph.mon.g.smithi173.stderr: 16: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.340 INFO:tasks.ceph.mon.g.smithi173.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55b6f57175bf] 2021-03-09T09:36:21.340 INFO:tasks.ceph.mon.g.smithi173.stderr: 18: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.341 INFO:tasks.ceph.mon.g.smithi173.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7f2a77cc03f5] 2021-03-09T09:36:21.341 INFO:tasks.ceph.mon.g.smithi173.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2a76acf6db] 2021-03-09T09:36:21.341 INFO:tasks.ceph.mon.g.smithi173.stderr: 21: clone() 2021-03-09T09:36:21.341 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.341 INFO:tasks.ceph.mon.g.smithi173.stderr: 0> 2021-03-09T09:36:21.276+0000 7f2a69c9a700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:36:21.342 INFO:tasks.ceph.mon.g.smithi173.stderr: in thread 7f2a69c9a700 thread_name:fn_monstore 2021-03-09T09:36:21.342 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.342 INFO:tasks.ceph.mon.g.smithi173.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:21.342 INFO:tasks.ceph.mon.g.smithi173.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f2a76ada980] 2021-03-09T09:36:21.343 INFO:tasks.ceph.mon.g.smithi173.stderr: 2: gsignal() 2021-03-09T09:36:21.343 INFO:tasks.ceph.mon.g.smithi173.stderr: 3: abort() 2021-03-09T09:36:21.343 INFO:tasks.ceph.mon.g.smithi173.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f2a77c5f42e] 2021-03-09T09:36:21.343 INFO:tasks.ceph.mon.g.smithi173.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f2a77c5f5b8] 2021-03-09T09:36:21.343 INFO:tasks.ceph.mon.g.smithi173.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55b6f5726a65] 2021-03-09T09:36:21.344 INFO:tasks.ceph.mon.g.smithi173.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55b6f5772a71] 2021-03-09T09:36:21.344 INFO:tasks.ceph.mon.g.smithi173.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55b6f57a2295] 2021-03-09T09:36:21.344 INFO:tasks.ceph.mon.g.smithi173.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55b6f57aa24b] 2021-03-09T09:36:21.344 INFO:tasks.ceph.mon.g.smithi173.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55b6f57aa7ab] 2021-03-09T09:36:21.344 INFO:tasks.ceph.mon.g.smithi173.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55b6f5719ade] 2021-03-09T09:36:21.345 INFO:tasks.ceph.mon.g.smithi173.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55b6f566f134] 2021-03-09T09:36:21.345 INFO:tasks.ceph.mon.g.smithi173.stderr: 13: (C_MonOp::finish(int)+0x45) [0x55b6f560daa5] 2021-03-09T09:36:21.345 INFO:tasks.ceph.mon.g.smithi173.stderr: 14: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.345 INFO:tasks.ceph.mon.g.smithi173.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55b6f5636548] 2021-03-09T09:36:21.345 INFO:tasks.ceph.mon.g.smithi173.stderr: 16: (Paxos::finish_round()+0x9b) [0x55b6f57117db] 2021-03-09T09:36:21.346 INFO:tasks.ceph.mon.g.smithi173.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x55b6f57140c0] 2021-03-09T09:36:21.346 INFO:tasks.ceph.mon.g.smithi173.stderr: 18: (C_Committed::finish(int)+0x41) [0x55b6f5717861] 2021-03-09T09:36:21.346 INFO:tasks.ceph.mon.g.smithi173.stderr: 19: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.346 INFO:tasks.ceph.mon.g.smithi173.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55b6f57175bf] 2021-03-09T09:36:21.347 INFO:tasks.ceph.mon.g.smithi173.stderr: 21: (Context::complete(int)+0x9) [0x55b6f560b059] 2021-03-09T09:36:21.347 INFO:tasks.ceph.mon.g.smithi173.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7f2a77cc03f5] 2021-03-09T09:36:21.347 INFO:tasks.ceph.mon.g.smithi173.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f2a76acf6db] 2021-03-09T09:36:21.347 INFO:tasks.ceph.mon.g.smithi173.stderr: 24: clone() 2021-03-09T09:36:21.348 INFO:tasks.ceph.mon.g.smithi173.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:36:21.348 INFO:tasks.ceph.mon.g.smithi173.stderr: 2021-03-09T09:36:21.671 INFO:tasks.ceph.mon.c.smithi143.stderr:2021-03-09T09:36:21.668+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:21.682 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:21.676+0000 7fd6e9553700 -1 mon.e@8(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:21.684 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:21.676+0000 7f445d90b700 -1 mon.d@6(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:21.734 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:21.728+0000 7fbab2ccd700 -1 mon.i@7(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:21.737 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:21.732+0000 7fcabb99b700 -1 mon.h@5(peon) e1 get_health_metrics reporting 4 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:22.107 INFO:tasks.ceph.mon.g.smithi173.stderr:daemon-helper: command crashed with signal 6 2021-03-09T09:36:22.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:22.292+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:22.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:22.292+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:23.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:23.292+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:23.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:23.292+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:24.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:24.292+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:24.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:24.292+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:25.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:25.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:25.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:25.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:25.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:25.296+0000 7ff35e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:26.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:26.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:26.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:26.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:26.672 INFO:tasks.ceph.mon.c.smithi143.stderr:2021-03-09T09:36:26.664+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:26.682 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:26.676+0000 7fd6e9553700 -1 mon.e@8(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:26.684 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:26.680+0000 7f445d90b700 -1 mon.d@6(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:26.734 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:26.732+0000 7fbab2ccd700 -1 mon.i@7(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:26.738 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:26.732+0000 7fcabb99b700 -1 mon.h@5(peon) e1 get_health_metrics reporting 4 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:27.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:27.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:27.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:27.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:28.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:28.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:28.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:28.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:29.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:29.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:29.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:29.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:30.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:30.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:30.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:30.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:31.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:31.292+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:31.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:31.292+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:31.672 INFO:tasks.ceph.mon.c.smithi143.stderr:2021-03-09T09:36:31.668+0000 7f4e89a2e700 -1 mon.c@4(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:31.682 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:31.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:31.685 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:31.680+0000 7f445d90b700 -1 mon.d@6(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:31.734 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:31.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:31.738 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:31.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:32.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:32.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:32.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:32.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:33.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:33.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:33.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:33.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:34.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:34.292+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:34.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:34.292+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:35.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:35.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:35.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:35.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:35.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:35.296+0000 7ff35e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:36.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:36.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:36.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:36.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:36.672 INFO:tasks.ceph.mon.c.smithi143.stderr:2021-03-09T09:36:36.668+0000 7f4e89a2e700 -1 mon.c@4(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:36.682 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:36.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:36.684 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:36.680+0000 7f445d90b700 -1 mon.d@6(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:36.734 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:36.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:36.738 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:36.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:37.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:37.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:37.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:38.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:38.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:38.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:38.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:39.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:39.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:39.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:39.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:40.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:40.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:40.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:40.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:41.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:41.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:41.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:41.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:41.312 INFO:tasks.ceph.mon.c.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f4e85a26700 time 2021-03-09T09:36:41.312057+0000 2021-03-09T09:36:41.312 INFO:tasks.ceph.mon.c.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:41.326 INFO:tasks.ceph.mon.c.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:41.326 INFO:tasks.ceph.mon.c.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f4e939eb3dd] 2021-03-09T09:36:41.326 INFO:tasks.ceph.mon.c.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4e939eb5b8] 2021-03-09T09:36:41.326 INFO:tasks.ceph.mon.c.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55607f91ca65] 2021-03-09T09:36:41.327 INFO:tasks.ceph.mon.c.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55607f968a71] 2021-03-09T09:36:41.327 INFO:tasks.ceph.mon.c.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55607f998295] 2021-03-09T09:36:41.327 INFO:tasks.ceph.mon.c.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55607f9a024b] 2021-03-09T09:36:41.327 INFO:tasks.ceph.mon.c.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55607f9a07ab] 2021-03-09T09:36:41.328 INFO:tasks.ceph.mon.c.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55607f90fade] 2021-03-09T09:36:41.328 INFO:tasks.ceph.mon.c.smithi143.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55607f865134] 2021-03-09T09:36:41.328 INFO:tasks.ceph.mon.c.smithi143.stderr: 10: (C_MonOp::finish(int)+0x45) [0x55607f803aa5] 2021-03-09T09:36:41.328 INFO:tasks.ceph.mon.c.smithi143.stderr: 11: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.328 INFO:tasks.ceph.mon.c.smithi143.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55607f82c548] 2021-03-09T09:36:41.329 INFO:tasks.ceph.mon.c.smithi143.stderr: 13: (Paxos::finish_round()+0x9b) [0x55607f9077db] 2021-03-09T09:36:41.329 INFO:tasks.ceph.mon.c.smithi143.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x55607f90a0c0] 2021-03-09T09:36:41.329 INFO:tasks.ceph.mon.c.smithi143.stderr: 15: (C_Committed::finish(int)+0x41) [0x55607f90d861] 2021-03-09T09:36:41.329 INFO:tasks.ceph.mon.c.smithi143.stderr: 16: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.329 INFO:tasks.ceph.mon.c.smithi143.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55607f90d5bf] 2021-03-09T09:36:41.330 INFO:tasks.ceph.mon.c.smithi143.stderr: 18: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.330 INFO:tasks.ceph.mon.c.smithi143.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7f4e93a4c3f5] 2021-03-09T09:36:41.330 INFO:tasks.ceph.mon.c.smithi143.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4e9285b6db] 2021-03-09T09:36:41.330 INFO:tasks.ceph.mon.c.smithi143.stderr: 21: clone() 2021-03-09T09:36:41.331 INFO:tasks.ceph.mon.c.smithi143.stderr:*** Caught signal (Aborted) ** 2021-03-09T09:36:41.331 INFO:tasks.ceph.mon.c.smithi143.stderr: in thread 7f4e85a26700 thread_name:fn_monstore 2021-03-09T09:36:41.331 INFO:tasks.ceph.mon.c.smithi143.stderr:2021-03-09T09:36:41.320+0000 7f4e85a26700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f4e85a26700 time 2021-03-09T09:36:41.312057+0000 2021-03-09T09:36:41.331 INFO:tasks.ceph.mon.c.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:41.332 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.332 INFO:tasks.ceph.mon.c.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:41.332 INFO:tasks.ceph.mon.c.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f4e939eb3dd] 2021-03-09T09:36:41.332 INFO:tasks.ceph.mon.c.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4e939eb5b8] 2021-03-09T09:36:41.332 INFO:tasks.ceph.mon.c.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55607f91ca65] 2021-03-09T09:36:41.333 INFO:tasks.ceph.mon.c.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55607f968a71] 2021-03-09T09:36:41.333 INFO:tasks.ceph.mon.c.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55607f998295] 2021-03-09T09:36:41.333 INFO:tasks.ceph.mon.c.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55607f9a024b] 2021-03-09T09:36:41.333 INFO:tasks.ceph.mon.c.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55607f9a07ab] 2021-03-09T09:36:41.333 INFO:tasks.ceph.mon.c.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55607f90fade] 2021-03-09T09:36:41.334 INFO:tasks.ceph.mon.c.smithi143.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55607f865134] 2021-03-09T09:36:41.334 INFO:tasks.ceph.mon.c.smithi143.stderr: 10: (C_MonOp::finish(int)+0x45) [0x55607f803aa5] 2021-03-09T09:36:41.334 INFO:tasks.ceph.mon.c.smithi143.stderr: 11: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.335 INFO:tasks.ceph.mon.c.smithi143.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55607f82c548] 2021-03-09T09:36:41.335 INFO:tasks.ceph.mon.c.smithi143.stderr: 13: (Paxos::finish_round()+0x9b) [0x55607f9077db] 2021-03-09T09:36:41.335 INFO:tasks.ceph.mon.c.smithi143.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x55607f90a0c0] 2021-03-09T09:36:41.335 INFO:tasks.ceph.mon.c.smithi143.stderr: 15: (C_Committed::finish(int)+0x41) [0x55607f90d861] 2021-03-09T09:36:41.335 INFO:tasks.ceph.mon.c.smithi143.stderr: 16: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.336 INFO:tasks.ceph.mon.c.smithi143.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55607f90d5bf] 2021-03-09T09:36:41.336 INFO:tasks.ceph.mon.c.smithi143.stderr: 18: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.337 INFO:tasks.ceph.mon.c.smithi143.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7f4e93a4c3f5] 2021-03-09T09:36:41.337 INFO:tasks.ceph.mon.c.smithi143.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4e9285b6db] 2021-03-09T09:36:41.337 INFO:tasks.ceph.mon.c.smithi143.stderr: 21: clone() 2021-03-09T09:36:41.337 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.339 INFO:tasks.ceph.mon.c.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:41.339 INFO:tasks.ceph.mon.c.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f4e92866980] 2021-03-09T09:36:41.339 INFO:tasks.ceph.mon.c.smithi143.stderr: 2: gsignal() 2021-03-09T09:36:41.339 INFO:tasks.ceph.mon.c.smithi143.stderr: 3: abort() 2021-03-09T09:36:41.339 INFO:tasks.ceph.mon.c.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f4e939eb42e] 2021-03-09T09:36:41.340 INFO:tasks.ceph.mon.c.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4e939eb5b8] 2021-03-09T09:36:41.340 INFO:tasks.ceph.mon.c.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55607f91ca65] 2021-03-09T09:36:41.340 INFO:tasks.ceph.mon.c.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55607f968a71] 2021-03-09T09:36:41.341 INFO:tasks.ceph.mon.c.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55607f998295] 2021-03-09T09:36:41.341 INFO:tasks.ceph.mon.c.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55607f9a024b] 2021-03-09T09:36:41.341 INFO:tasks.ceph.mon.c.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55607f9a07ab] 2021-03-09T09:36:41.341 INFO:tasks.ceph.mon.c.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55607f90fade] 2021-03-09T09:36:41.341 INFO:tasks.ceph.mon.c.smithi143.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55607f865134] 2021-03-09T09:36:41.342 INFO:tasks.ceph.mon.c.smithi143.stderr: 13: (C_MonOp::finish(int)+0x45) [0x55607f803aa5] 2021-03-09T09:36:41.342 INFO:tasks.ceph.mon.c.smithi143.stderr: 14: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.342 INFO:tasks.ceph.mon.c.smithi143.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55607f82c548] 2021-03-09T09:36:41.342 INFO:tasks.ceph.mon.c.smithi143.stderr: 16: (Paxos::finish_round()+0x9b) [0x55607f9077db] 2021-03-09T09:36:41.343 INFO:tasks.ceph.mon.c.smithi143.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x55607f90a0c0] 2021-03-09T09:36:41.343 INFO:tasks.ceph.mon.c.smithi143.stderr: 18: (C_Committed::finish(int)+0x41) [0x55607f90d861] 2021-03-09T09:36:41.343 INFO:tasks.ceph.mon.c.smithi143.stderr: 19: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.343 INFO:tasks.ceph.mon.c.smithi143.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55607f90d5bf] 2021-03-09T09:36:41.343 INFO:tasks.ceph.mon.c.smithi143.stderr: 21: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.344 INFO:tasks.ceph.mon.c.smithi143.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7f4e93a4c3f5] 2021-03-09T09:36:41.344 INFO:tasks.ceph.mon.c.smithi143.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4e9285b6db] 2021-03-09T09:36:41.344 INFO:tasks.ceph.mon.c.smithi143.stderr: 24: clone() 2021-03-09T09:36:41.344 INFO:tasks.ceph.mon.c.smithi143.stderr:2021-03-09T09:36:41.328+0000 7f4e85a26700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:36:41.344 INFO:tasks.ceph.mon.c.smithi143.stderr: in thread 7f4e85a26700 thread_name:fn_monstore 2021-03-09T09:36:41.345 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.345 INFO:tasks.ceph.mon.c.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:41.345 INFO:tasks.ceph.mon.c.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f4e92866980] 2021-03-09T09:36:41.345 INFO:tasks.ceph.mon.c.smithi143.stderr: 2: gsignal() 2021-03-09T09:36:41.345 INFO:tasks.ceph.mon.c.smithi143.stderr: 3: abort() 2021-03-09T09:36:41.346 INFO:tasks.ceph.mon.c.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f4e939eb42e] 2021-03-09T09:36:41.346 INFO:tasks.ceph.mon.c.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4e939eb5b8] 2021-03-09T09:36:41.346 INFO:tasks.ceph.mon.c.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55607f91ca65] 2021-03-09T09:36:41.346 INFO:tasks.ceph.mon.c.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55607f968a71] 2021-03-09T09:36:41.347 INFO:tasks.ceph.mon.c.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55607f998295] 2021-03-09T09:36:41.347 INFO:tasks.ceph.mon.c.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55607f9a024b] 2021-03-09T09:36:41.347 INFO:tasks.ceph.mon.c.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55607f9a07ab] 2021-03-09T09:36:41.347 INFO:tasks.ceph.mon.c.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55607f90fade] 2021-03-09T09:36:41.347 INFO:tasks.ceph.mon.c.smithi143.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55607f865134] 2021-03-09T09:36:41.348 INFO:tasks.ceph.mon.c.smithi143.stderr: 13: (C_MonOp::finish(int)+0x45) [0x55607f803aa5] 2021-03-09T09:36:41.348 INFO:tasks.ceph.mon.c.smithi143.stderr: 14: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.348 INFO:tasks.ceph.mon.c.smithi143.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55607f82c548] 2021-03-09T09:36:41.348 INFO:tasks.ceph.mon.c.smithi143.stderr: 16: (Paxos::finish_round()+0x9b) [0x55607f9077db] 2021-03-09T09:36:41.348 INFO:tasks.ceph.mon.c.smithi143.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x55607f90a0c0] 2021-03-09T09:36:41.349 INFO:tasks.ceph.mon.c.smithi143.stderr: 18: (C_Committed::finish(int)+0x41) [0x55607f90d861] 2021-03-09T09:36:41.349 INFO:tasks.ceph.mon.c.smithi143.stderr: 19: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.349 INFO:tasks.ceph.mon.c.smithi143.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55607f90d5bf] 2021-03-09T09:36:41.349 INFO:tasks.ceph.mon.c.smithi143.stderr: 21: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.349 INFO:tasks.ceph.mon.c.smithi143.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7f4e93a4c3f5] 2021-03-09T09:36:41.350 INFO:tasks.ceph.mon.c.smithi143.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4e9285b6db] 2021-03-09T09:36:41.350 INFO:tasks.ceph.mon.c.smithi143.stderr: 24: clone() 2021-03-09T09:36:41.350 INFO:tasks.ceph.mon.c.smithi143.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:36:41.350 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.352 INFO:tasks.ceph.mon.c.smithi143.stderr: -6280> 2021-03-09T09:36:06.664+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.352 INFO:tasks.ceph.mon.c.smithi143.stderr: -5614> 2021-03-09T09:36:11.666+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.352 INFO:tasks.ceph.mon.c.smithi143.stderr: -4640> 2021-03-09T09:36:16.668+0000 7f4e89a2e700 -1 mon.c@4(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.353 INFO:tasks.ceph.mon.c.smithi143.stderr: -3597> 2021-03-09T09:36:21.668+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.353 INFO:tasks.ceph.mon.c.smithi143.stderr: -3016> 2021-03-09T09:36:26.664+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.353 INFO:tasks.ceph.mon.c.smithi143.stderr: -2224> 2021-03-09T09:36:31.668+0000 7f4e89a2e700 -1 mon.c@4(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.353 INFO:tasks.ceph.mon.c.smithi143.stderr: -1607> 2021-03-09T09:36:36.668+0000 7f4e89a2e700 -1 mon.c@4(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.353 INFO:tasks.ceph.mon.c.smithi143.stderr: -1> 2021-03-09T09:36:41.320+0000 7f4e85a26700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f4e85a26700 time 2021-03-09T09:36:41.312057+0000 2021-03-09T09:36:41.354 INFO:tasks.ceph.mon.c.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:41.354 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.354 INFO:tasks.ceph.mon.c.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:41.354 INFO:tasks.ceph.mon.c.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f4e939eb3dd] 2021-03-09T09:36:41.355 INFO:tasks.ceph.mon.c.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4e939eb5b8] 2021-03-09T09:36:41.355 INFO:tasks.ceph.mon.c.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55607f91ca65] 2021-03-09T09:36:41.355 INFO:tasks.ceph.mon.c.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55607f968a71] 2021-03-09T09:36:41.355 INFO:tasks.ceph.mon.c.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55607f998295] 2021-03-09T09:36:41.356 INFO:tasks.ceph.mon.c.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55607f9a024b] 2021-03-09T09:36:41.356 INFO:tasks.ceph.mon.c.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55607f9a07ab] 2021-03-09T09:36:41.356 INFO:tasks.ceph.mon.c.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55607f90fade] 2021-03-09T09:36:41.356 INFO:tasks.ceph.mon.c.smithi143.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55607f865134] 2021-03-09T09:36:41.356 INFO:tasks.ceph.mon.c.smithi143.stderr: 10: (C_MonOp::finish(int)+0x45) [0x55607f803aa5] 2021-03-09T09:36:41.357 INFO:tasks.ceph.mon.c.smithi143.stderr: 11: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.357 INFO:tasks.ceph.mon.c.smithi143.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55607f82c548] 2021-03-09T09:36:41.357 INFO:tasks.ceph.mon.c.smithi143.stderr: 13: (Paxos::finish_round()+0x9b) [0x55607f9077db] 2021-03-09T09:36:41.357 INFO:tasks.ceph.mon.c.smithi143.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x55607f90a0c0] 2021-03-09T09:36:41.357 INFO:tasks.ceph.mon.c.smithi143.stderr: 15: (C_Committed::finish(int)+0x41) [0x55607f90d861] 2021-03-09T09:36:41.358 INFO:tasks.ceph.mon.c.smithi143.stderr: 16: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.358 INFO:tasks.ceph.mon.c.smithi143.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55607f90d5bf] 2021-03-09T09:36:41.358 INFO:tasks.ceph.mon.c.smithi143.stderr: 18: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.358 INFO:tasks.ceph.mon.c.smithi143.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7f4e93a4c3f5] 2021-03-09T09:36:41.359 INFO:tasks.ceph.mon.c.smithi143.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4e9285b6db] 2021-03-09T09:36:41.359 INFO:tasks.ceph.mon.c.smithi143.stderr: 21: clone() 2021-03-09T09:36:41.359 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.359 INFO:tasks.ceph.mon.c.smithi143.stderr: 0> 2021-03-09T09:36:41.328+0000 7f4e85a26700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:36:41.359 INFO:tasks.ceph.mon.c.smithi143.stderr: in thread 7f4e85a26700 thread_name:fn_monstore 2021-03-09T09:36:41.360 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.360 INFO:tasks.ceph.mon.c.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:41.360 INFO:tasks.ceph.mon.c.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f4e92866980] 2021-03-09T09:36:41.360 INFO:tasks.ceph.mon.c.smithi143.stderr: 2: gsignal() 2021-03-09T09:36:41.360 INFO:tasks.ceph.mon.c.smithi143.stderr: 3: abort() 2021-03-09T09:36:41.361 INFO:tasks.ceph.mon.c.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f4e939eb42e] 2021-03-09T09:36:41.361 INFO:tasks.ceph.mon.c.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4e939eb5b8] 2021-03-09T09:36:41.361 INFO:tasks.ceph.mon.c.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55607f91ca65] 2021-03-09T09:36:41.361 INFO:tasks.ceph.mon.c.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55607f968a71] 2021-03-09T09:36:41.362 INFO:tasks.ceph.mon.c.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55607f998295] 2021-03-09T09:36:41.362 INFO:tasks.ceph.mon.c.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55607f9a024b] 2021-03-09T09:36:41.362 INFO:tasks.ceph.mon.c.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55607f9a07ab] 2021-03-09T09:36:41.362 INFO:tasks.ceph.mon.c.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55607f90fade] 2021-03-09T09:36:41.362 INFO:tasks.ceph.mon.c.smithi143.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55607f865134] 2021-03-09T09:36:41.363 INFO:tasks.ceph.mon.c.smithi143.stderr: 13: (C_MonOp::finish(int)+0x45) [0x55607f803aa5] 2021-03-09T09:36:41.363 INFO:tasks.ceph.mon.c.smithi143.stderr: 14: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.363 INFO:tasks.ceph.mon.c.smithi143.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55607f82c548] 2021-03-09T09:36:41.363 INFO:tasks.ceph.mon.c.smithi143.stderr: 16: (Paxos::finish_round()+0x9b) [0x55607f9077db] 2021-03-09T09:36:41.364 INFO:tasks.ceph.mon.c.smithi143.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x55607f90a0c0] 2021-03-09T09:36:41.364 INFO:tasks.ceph.mon.c.smithi143.stderr: 18: (C_Committed::finish(int)+0x41) [0x55607f90d861] 2021-03-09T09:36:41.364 INFO:tasks.ceph.mon.c.smithi143.stderr: 19: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.364 INFO:tasks.ceph.mon.c.smithi143.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55607f90d5bf] 2021-03-09T09:36:41.364 INFO:tasks.ceph.mon.c.smithi143.stderr: 21: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.365 INFO:tasks.ceph.mon.c.smithi143.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7f4e93a4c3f5] 2021-03-09T09:36:41.365 INFO:tasks.ceph.mon.c.smithi143.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4e9285b6db] 2021-03-09T09:36:41.365 INFO:tasks.ceph.mon.c.smithi143.stderr: 24: clone() 2021-03-09T09:36:41.365 INFO:tasks.ceph.mon.c.smithi143.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:36:41.365 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.367 INFO:tasks.ceph.mon.c.smithi143.stderr: -6280> 2021-03-09T09:36:06.664+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.367 INFO:tasks.ceph.mon.c.smithi143.stderr: -5614> 2021-03-09T09:36:11.666+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.367 INFO:tasks.ceph.mon.c.smithi143.stderr: -4640> 2021-03-09T09:36:16.668+0000 7f4e89a2e700 -1 mon.c@4(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.368 INFO:tasks.ceph.mon.c.smithi143.stderr: -3597> 2021-03-09T09:36:21.668+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.368 INFO:tasks.ceph.mon.c.smithi143.stderr: -3016> 2021-03-09T09:36:26.664+0000 7f4e89a2e700 -1 mon.c@4(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.368 INFO:tasks.ceph.mon.c.smithi143.stderr: -2224> 2021-03-09T09:36:31.668+0000 7f4e89a2e700 -1 mon.c@4(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.368 INFO:tasks.ceph.mon.c.smithi143.stderr: -1607> 2021-03-09T09:36:36.668+0000 7f4e89a2e700 -1 mon.c@4(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 4 at 2021-03-09T09:35:32.079487+0000) 2021-03-09T09:36:41.369 INFO:tasks.ceph.mon.c.smithi143.stderr: -1> 2021-03-09T09:36:41.320+0000 7f4e85a26700 -1 /build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f4e85a26700 time 2021-03-09T09:36:41.312057+0000 2021-03-09T09:36:41.369 INFO:tasks.ceph.mon.c.smithi143.stderr:/build/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T09:36:41.369 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.369 INFO:tasks.ceph.mon.c.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:41.369 INFO:tasks.ceph.mon.c.smithi143.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f4e939eb3dd] 2021-03-09T09:36:41.370 INFO:tasks.ceph.mon.c.smithi143.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4e939eb5b8] 2021-03-09T09:36:41.370 INFO:tasks.ceph.mon.c.smithi143.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55607f91ca65] 2021-03-09T09:36:41.370 INFO:tasks.ceph.mon.c.smithi143.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55607f968a71] 2021-03-09T09:36:41.370 INFO:tasks.ceph.mon.c.smithi143.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55607f998295] 2021-03-09T09:36:41.371 INFO:tasks.ceph.mon.c.smithi143.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55607f9a024b] 2021-03-09T09:36:41.371 INFO:tasks.ceph.mon.c.smithi143.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55607f9a07ab] 2021-03-09T09:36:41.371 INFO:tasks.ceph.mon.c.smithi143.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55607f90fade] 2021-03-09T09:36:41.371 INFO:tasks.ceph.mon.c.smithi143.stderr: 9: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55607f865134] 2021-03-09T09:36:41.371 INFO:tasks.ceph.mon.c.smithi143.stderr: 10: (C_MonOp::finish(int)+0x45) [0x55607f803aa5] 2021-03-09T09:36:41.372 INFO:tasks.ceph.mon.c.smithi143.stderr: 11: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.372 INFO:tasks.ceph.mon.c.smithi143.stderr: 12: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55607f82c548] 2021-03-09T09:36:41.372 INFO:tasks.ceph.mon.c.smithi143.stderr: 13: (Paxos::finish_round()+0x9b) [0x55607f9077db] 2021-03-09T09:36:41.372 INFO:tasks.ceph.mon.c.smithi143.stderr: 14: (Paxos::commit_finish()+0x7e0) [0x55607f90a0c0] 2021-03-09T09:36:41.372 INFO:tasks.ceph.mon.c.smithi143.stderr: 15: (C_Committed::finish(int)+0x41) [0x55607f90d861] 2021-03-09T09:36:41.373 INFO:tasks.ceph.mon.c.smithi143.stderr: 16: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.373 INFO:tasks.ceph.mon.c.smithi143.stderr: 17: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55607f90d5bf] 2021-03-09T09:36:41.373 INFO:tasks.ceph.mon.c.smithi143.stderr: 18: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.373 INFO:tasks.ceph.mon.c.smithi143.stderr: 19: (Finisher::finisher_thread_entry()+0x195) [0x7f4e93a4c3f5] 2021-03-09T09:36:41.374 INFO:tasks.ceph.mon.c.smithi143.stderr: 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4e9285b6db] 2021-03-09T09:36:41.374 INFO:tasks.ceph.mon.c.smithi143.stderr: 21: clone() 2021-03-09T09:36:41.374 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.374 INFO:tasks.ceph.mon.c.smithi143.stderr: 0> 2021-03-09T09:36:41.328+0000 7f4e85a26700 -1 *** Caught signal (Aborted) ** 2021-03-09T09:36:41.374 INFO:tasks.ceph.mon.c.smithi143.stderr: in thread 7f4e85a26700 thread_name:fn_monstore 2021-03-09T09:36:41.375 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.375 INFO:tasks.ceph.mon.c.smithi143.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T09:36:41.375 INFO:tasks.ceph.mon.c.smithi143.stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f4e92866980] 2021-03-09T09:36:41.375 INFO:tasks.ceph.mon.c.smithi143.stderr: 2: gsignal() 2021-03-09T09:36:41.375 INFO:tasks.ceph.mon.c.smithi143.stderr: 3: abort() 2021-03-09T09:36:41.376 INFO:tasks.ceph.mon.c.smithi143.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f4e939eb42e] 2021-03-09T09:36:41.376 INFO:tasks.ceph.mon.c.smithi143.stderr: 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f4e939eb5b8] 2021-03-09T09:36:41.376 INFO:tasks.ceph.mon.c.smithi143.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4f5) [0x55607f91ca65] 2021-03-09T09:36:41.376 INFO:tasks.ceph.mon.c.smithi143.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x551) [0x55607f968a71] 2021-03-09T09:36:41.377 INFO:tasks.ceph.mon.c.smithi143.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1e4c5) [0x55607f998295] 2021-03-09T09:36:41.377 INFO:tasks.ceph.mon.c.smithi143.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0x10b) [0x55607f9a024b] 2021-03-09T09:36:41.377 INFO:tasks.ceph.mon.c.smithi143.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x26b) [0x55607f9a07ab] 2021-03-09T09:36:41.377 INFO:tasks.ceph.mon.c.smithi143.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0x9be) [0x55607f90fade] 2021-03-09T09:36:41.377 INFO:tasks.ceph.mon.c.smithi143.stderr: 12: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x55607f865134] 2021-03-09T09:36:41.378 INFO:tasks.ceph.mon.c.smithi143.stderr: 13: (C_MonOp::finish(int)+0x45) [0x55607f803aa5] 2021-03-09T09:36:41.378 INFO:tasks.ceph.mon.c.smithi143.stderr: 14: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.378 INFO:tasks.ceph.mon.c.smithi143.stderr: 15: (void finish_contexts > >(ceph::common::CephContext*, std::__cxx11::list >&, int)+0xa8) [0x55607f82c548] 2021-03-09T09:36:41.378 INFO:tasks.ceph.mon.c.smithi143.stderr: 16: (Paxos::finish_round()+0x9b) [0x55607f9077db] 2021-03-09T09:36:41.378 INFO:tasks.ceph.mon.c.smithi143.stderr: 17: (Paxos::commit_finish()+0x7e0) [0x55607f90a0c0] 2021-03-09T09:36:41.379 INFO:tasks.ceph.mon.c.smithi143.stderr: 18: (C_Committed::finish(int)+0x41) [0x55607f90d861] 2021-03-09T09:36:41.379 INFO:tasks.ceph.mon.c.smithi143.stderr: 19: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.379 INFO:tasks.ceph.mon.c.smithi143.stderr: 20: (MonitorDBStore::C_DoTransaction::finish(int)+0x8f) [0x55607f90d5bf] 2021-03-09T09:36:41.379 INFO:tasks.ceph.mon.c.smithi143.stderr: 21: (Context::complete(int)+0x9) [0x55607f801059] 2021-03-09T09:36:41.380 INFO:tasks.ceph.mon.c.smithi143.stderr: 22: (Finisher::finisher_thread_entry()+0x195) [0x7f4e93a4c3f5] 2021-03-09T09:36:41.380 INFO:tasks.ceph.mon.c.smithi143.stderr: 23: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f4e9285b6db] 2021-03-09T09:36:41.380 INFO:tasks.ceph.mon.c.smithi143.stderr: 24: clone() 2021-03-09T09:36:41.380 INFO:tasks.ceph.mon.c.smithi143.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T09:36:41.380 INFO:tasks.ceph.mon.c.smithi143.stderr: 2021-03-09T09:36:41.683 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:41.680+0000 7fd6e9553700 -1 mon.e@8(peon) e1 get_health_metrics reporting 3 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:41.862 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:41.728+0000 7fbab2ccd700 -1 mon.i@7(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:41.863 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:41.680+0000 7f445d90b700 -1 mon.d@6(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:41.863 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:41.732+0000 7fcabb99b700 -1 mon.h@5(peon) e1 get_health_metrics reporting 5 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:42.079 INFO:tasks.ceph.mon.c.smithi143.stderr:daemon-helper: command crashed with signal 6 2021-03-09T09:36:42.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:42.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:42.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:42.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:43.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:43.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:43.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:43.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:44.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:44.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:44.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:44.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:45.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:45.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:45.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:45.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:45.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:45.300+0000 7ff35e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:46.301 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:46.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:46.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:46.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:46.683 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:46.680+0000 7fd6e9553700 -1 mon.e@8(peon) e1 get_health_metrics reporting 4 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:46.685 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:46.680+0000 7f445d90b700 -1 mon.d@6(peon) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:46.734 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:46.732+0000 7fbab2ccd700 -1 mon.i@7(peon) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:46.739 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:46.732+0000 7fcabb99b700 -1 mon.h@5(peon) e1 get_health_metrics reporting 6 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:47.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:47.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:47.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:47.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:48.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:48.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:48.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:48.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:49.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:49.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:49.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:49.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:50.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:50.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:50.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:51.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:51.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:51.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:51.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:51.683 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:51.676+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 4 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:51.685 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:51.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:51.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:51.728+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:51.739 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:51.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:52.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:52.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:52.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:52.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:53.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:53.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:53.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:53.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:54.302 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:54.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:54.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:54.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:55.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:55.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:55.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:55.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:55.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:55.300+0000 7ff35e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:56.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:56.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:56.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:56.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:56.683 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:36:56.676+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 4 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.665353+0000) 2021-03-09T09:36:56.685 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:36:56.680+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:36:56.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:36:56.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:36:56.739 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:36:56.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(2 entries from seq 1 at 2021-03-09T09:35:29.712155+0000) 2021-03-09T09:36:57.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:57.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:57.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:57.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:58.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:58.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:58.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:58.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:59.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:59.300+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:36:59.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:36:59.300+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:00.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:00.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:37:00.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:00.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:01.303 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.296+0000 7ff35e7fc700 10 monclient: tick 2021-03-09T09:37:01.304 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.296+0000 7ff35e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:01.683 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:01.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:01.685 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:01.680+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:01.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:01.728+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:01.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:01.732+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:01.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff37412dce0 legacy=0x7ff374142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:01.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff37412dce0 legacy=0x7ff374142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:01.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff37412dce0 0x7ff374142b80 :-1 s=OPENED pgs=58 cs=1 l=1).handle_message read tag failed 2021-03-09T09:37:01.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff37412dce0 0x7ff374142b80 :-1 s=OPENED pgs=58 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:37:01.742 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 10 monclient: ms_handle_reset current mon v1:172.21.15.173:6791/0 2021-03-09T09:37:01.742 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:01.742 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff37412dce0 legacy=0x7ff374142b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:01.742 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 10 monclient: _add_conns ranks=[4,6,8,5,1,7,2,0,3] 2021-03-09T09:37:01.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 10 monclient(hunting): picked mon.c con 0x7ff3540365e0 addr v1:172.21.15.143:6791/0 2021-03-09T09:37:01.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 10 monclient(hunting): picked mon.d con 0x7ff35403a190 addr v1:172.21.15.143:6792/0 2021-03-09T09:37:01.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 10 monclient(hunting): picked mon.e con 0x7ff35403dec0 addr v1:172.21.15.143:6793/0 2021-03-09T09:37:01.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3540365e0 legacy=0x7ff3540389d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:37:01.744 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff374066b90 con 0x7ff3540365e0 2021-03-09T09:37:01.744 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff37413b3e0 con 0x7ff35403a190 2021-03-09T09:37:01.744 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff374067730 con 0x7ff35403dec0 2021-03-09T09:37:01.744 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.736+0000 7ff35f7fe700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:01.941 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:01.940+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3540365e0 legacy=0x7ff3540389d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:37:02.304 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:37:02.304 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:37:02.305 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:37:02.305 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35403dec0 legacy=0x7ff354040380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:37:02.305 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35403a190 legacy=0x7ff35403c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:37:02.305 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3540365e0 legacy=0x7ff3540389d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:02.306 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,3,5,7,1,0] 2021-03-09T09:37:02.306 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:37:02.306 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500034a0 addr v1:172.21.15.143:6790/0 2021-03-09T09:37:02.306 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.296+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350007130 addr v1:172.21.15.173:6790/0 2021-03-09T09:37:02.306 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.300+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:37:02.307 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.300+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:02.307 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.300+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000ae90 addr v1:172.21.15.173:6791/0 2021-03-09T09:37:02.307 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.300+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff354044ca0 con 0x7ff3500034a0 2021-03-09T09:37:02.307 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.300+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff374066b90 con 0x7ff350007130 2021-03-09T09:37:02.308 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.300+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350007130 legacy=0x7ff3500095c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:02.308 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.300+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010c50 con 0x7ff35000ae90 2021-03-09T09:37:02.308 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.300+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:02.508 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.504+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:02.508 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.504+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350007130 legacy=0x7ff3500095c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:02.912 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.908+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:02.912 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:02.908+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350007130 legacy=0x7ff3500095c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:03.716 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:03.712+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:03.717 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:03.712+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350007130 legacy=0x7ff3500095c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:05.318 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:05.316+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:05.318 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:05.316+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350007130 legacy=0x7ff3500095c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:06.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:06.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:06.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:06.680+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:06.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:06.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:06.739 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:06.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:06.740 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:06.736+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ae90 legacy=0x7ff35000d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:06.740 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:06.736+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ae90 legacy=0x7ff35000d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:06.736+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ae90 0x7ff35000d330 :-1 s=OPENED pgs=106 cs=1 l=1).handle_message read tag failed 2021-03-09T09:37:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:06.736+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ae90 0x7ff35000d330 :-1 s=OPENED pgs=106 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:37:06.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:06.736+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:37:08.521 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:08.520+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:08.521 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:08.520+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350007130 legacy=0x7ff3500095c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:09.055 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:37:09.056 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:37:09.056 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:37:09.056 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ae90 legacy=0x7ff35000d330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:09.057 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350007130 legacy=0x7ff3500095c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:09.057 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:09.057 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,1,7] 2021-03-09T09:37:09.058 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350010e10 addr v1:172.21.15.143:6789/0 2021-03-09T09:37:09.058 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350013200 addr v1:172.21.15.173:6789/0 2021-03-09T09:37:09.058 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350016d10 addr v1:172.21.15.173:6792/0 2021-03-09T09:37:09.059 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011ea0 con 0x7ff350010e10 2021-03-09T09:37:09.059 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010e10 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:37:09.059 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff354044ca0 con 0x7ff350013200 2021-03-09T09:37:09.060 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff374066b90 con 0x7ff350016d10 2021-03-09T09:37:09.060 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:09.060 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.052+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013200 legacy=0x7ff3500155f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:09.259 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.256+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010e10 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:37:09.260 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.256+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013200 legacy=0x7ff3500155f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:09.660 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.660+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010e10 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:37:09.661 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:09.660+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013200 legacy=0x7ff3500155f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:10.467 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:10.464+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010e10 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:37:10.468 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:10.464+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013200 legacy=0x7ff3500155f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:11.683 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:11.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:11.685 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:11.680+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:11.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:11.728+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:11.736 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:11.732+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350016d10 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:37:11.736 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:11.732+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350016d10 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:11.736 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:11.732+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350016d10 0x7ff350019100 :-1 s=OPENED pgs=98 cs=1 l=1).handle_message read tag failed 2021-03-09T09:37:11.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:11.732+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350016d10 0x7ff350019100 :-1 s=OPENED pgs=98 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:37:11.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:11.732+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:37:11.739 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:11.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:12.068+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010e10 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:37:12.072 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:12.068+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013200 legacy=0x7ff3500155f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:15.275 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:15.272+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010e10 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:37:15.275 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:15.272+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013200 legacy=0x7ff3500155f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:16.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:16.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:16.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:16.680+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:16.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:16.728+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:16.739 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:16.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:19.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.172+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:37:19.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.172+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:37:19.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.172+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:37:19.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350016d10 legacy=0x7ff350019100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:19.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013200 legacy=0x7ff3500155f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:19.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010e10 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:19.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,1,6,8,0,5,3,7,4] 2021-03-09T09:37:19.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:37:19.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500034a0 addr v1:172.21.15.143:6790/0 2021-03-09T09:37:19.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:37:19.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350007130 addr v1:172.21.15.173:6789/0 2021-03-09T09:37:19.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35000ae90 addr v1:172.21.15.143:6792/0 2021-03-09T09:37:19.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:19.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d920 con 0x7ff350007130 2021-03-09T09:37:19.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011ea0 con 0x7ff3500034a0 2021-03-09T09:37:19.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff354044ca0 con 0x7ff35000ae90 2021-03-09T09:37:19.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:19.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.176+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350007130 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:19.383 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.380+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:19.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.380+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350007130 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:19.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.784+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:19.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:19.784+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350007130 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:20.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:20.588+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:20.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:20.588+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350007130 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:21.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:21.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:21.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:21.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:21.687 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:21.684+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae90 legacy=0x7ff35000d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:37:21.688 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:21.684+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae90 legacy=0x7ff35000d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:21.688 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:21.684+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae90 0x7ff35000d2d0 :-1 s=OPENED pgs=108 cs=1 l=1).handle_message read tag failed 2021-03-09T09:37:21.688 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:21.684+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae90 0x7ff35000d2d0 :-1 s=OPENED pgs=108 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:37:21.689 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:21.684+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:37:21.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:21.728+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 5 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:21.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:21.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:22.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:22.192+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:22.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:22.192+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350007130 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:25.397 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:25.396+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350007130 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:25.398 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:25.396+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:26.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:26.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:26.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:26.680+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:26.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:26.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:26.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:26.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:31.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:31.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:31.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:31.680+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:31.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:31.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:31.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:31.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:31.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:31.800+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:37:31.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:31.800+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350007130 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:37:34.367 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.360+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:37:34.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.360+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:37:34.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.360+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:37:34.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae90 legacy=0x7ff35000d2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:34.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500034a0 legacy=0x7ff3500058f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:34.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350007130 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:34.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,8,3,4,5,0] 2021-03-09T09:37:34.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350009870 addr v1:172.21.15.173:6792/0 2021-03-09T09:37:34.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001c020 addr v1:172.21.15.143:6793/0 2021-03-09T09:37:34.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35001c450 addr v1:172.21.15.173:6790/0 2021-03-09T09:37:34.370 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009680 con 0x7ff35001c450 2021-03-09T09:37:34.370 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d920 con 0x7ff350009870 2021-03-09T09:37:34.370 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011ea0 con 0x7ff35001c020 2021-03-09T09:37:34.370 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:34.371 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.364+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:34.568 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.568+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:34.975 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:34.972+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:35.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:35.776+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:36.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:36.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:36.685 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.680+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 legacy=0x7ff350006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:36.685 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.680+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 legacy=0x7ff350006000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:36.685 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.680+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 0x7ff350006000 :-1 s=OPENED pgs=125 cs=1 l=1).handle_message read tag failed 2021-03-09T09:37:36.685 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.680+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 0x7ff350006000 :-1 s=OPENED pgs=125 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:37:36.686 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.680+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:37:36.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:36.680+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:36.735 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:36.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:36.736 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.732+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350009870 legacy=0x7ff35001b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:36.736 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.732+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350009870 legacy=0x7ff35001b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:36.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.732+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350009870 0x7ff35001b910 :-1 s=OPENED pgs=111 cs=1 l=1).handle_message read tag failed 2021-03-09T09:37:36.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.732+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350009870 0x7ff35001b910 :-1 s=OPENED pgs=111 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:37:36.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.732+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:37:36.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:36.732+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:37:36.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:36.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:37.382 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:37.380+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:40.586 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:40.584+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:41.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:41.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:41.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:41.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:41.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:41.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:41.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:41.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:46.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:46.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:46.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:46.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:46.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:46.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:46.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:46.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:46.990 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:46.988+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:37:51.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:51.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:51.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:51.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:51.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:51.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:51.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:51.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:56.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:37:56.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:37:56.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:37:56.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:37:56.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:37:56.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:37:56.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:37:56.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:37:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:37:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:37:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:37:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 legacy=0x7ff350006000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350009870 legacy=0x7ff35001b910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,4,5] 2021-03-09T09:37:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350009870 addr v1:172.21.15.143:6789/0 2021-03-09T09:37:57.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001c020 addr v1:172.21.15.143:6791/0 2021-03-09T09:37:57.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500034a0 addr v1:172.21.15.173:6791/0 2021-03-09T09:37:57.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500068e0 con 0x7ff350009870 2021-03-09T09:37:57.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009680 con 0x7ff35001c020 2021-03-09T09:37:57.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011cd0 con 0x7ff3500034a0 2021-03-09T09:37:57.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:57.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:37:57.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.148+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:37:57.354 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.352+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:37:57.355 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.352+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:37:57.755 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.756+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:37:57.755 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:57.756+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:37:58.562 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:58.560+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:37:58.562 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:37:58.560+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:00.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:00.164+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:00.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:00.164+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:01.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:01.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:01.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:01.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:01.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:01.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:01.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:01.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:01.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:01.736+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500034a0 legacy=0x7ff350006ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:38:01.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:01.736+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500034a0 legacy=0x7ff350006ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:01.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:01.736+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500034a0 0x7ff350006ad0 :-1 s=OPENED pgs=131 cs=1 l=1).handle_message read tag failed 2021-03-09T09:38:01.742 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:01.736+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500034a0 0x7ff350006ad0 :-1 s=OPENED pgs=131 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:38:01.742 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:01.736+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:38:03.367 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:03.368+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:03.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:03.368+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:06.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:06.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:06.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:06.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:06.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:06.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:06.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:06.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:09.774 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:09.772+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:09.774 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:09.772+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:11.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:11.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:11.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:11.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:11.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:11.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:11.740 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:11.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:16.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:16.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:16.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:16.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:16.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:16.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:16.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:16.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:21.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:21.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:21.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:21.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:21.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:21.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:21.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:21.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:22.586 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:22.588+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:22.587 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:22.588+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:26.684 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:26.680+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:26.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:26.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:26.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:26.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:26.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:26.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:38:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:38:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:38:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500034a0 legacy=0x7ff350006ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,7,0,8,2,1,5,6,3] 2021-03-09T09:38:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001c020 addr v1:172.21.15.143:6791/0 2021-03-09T09:38:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35000ce60 addr v1:172.21.15.173:6792/0 2021-03-09T09:38:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350009870 addr v1:172.21.15.143:6789/0 2021-03-09T09:38:27.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071e0 con 0x7ff350009870 2021-03-09T09:38:27.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:27.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500068e0 con 0x7ff35001c020 2021-03-09T09:38:27.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009680 con 0x7ff35000ce60 2021-03-09T09:38:27.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:27.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.144+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:27.348 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.348+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:27.349 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.348+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:27.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.752+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:27.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:27.752+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:28.557 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:28.556+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:28.558 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:28.556+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:30.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:30.160+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:30.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:30.160+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:31.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:31.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:31.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:31.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:31.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:31.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:31.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:31.736+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce60 legacy=0x7ff35000d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:38:31.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:31.736+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce60 legacy=0x7ff35000d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:31.738 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:31.736+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce60 0x7ff35000d240 :-1 s=OPENED pgs=138 cs=1 l=1).handle_message read tag failed 2021-03-09T09:38:31.738 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:31.736+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce60 0x7ff35000d240 :-1 s=OPENED pgs=138 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:38:31.738 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:31.736+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:38:31.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:31.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:33.362 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:33.364+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:33.363 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:33.364+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:36.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:36.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:36.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:36.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:36.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:36.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:36.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:36.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:39.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:39.768+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:39.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:39.768+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:41.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:41.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:41.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:41.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:41.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:41.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:41.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:41.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:46.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:46.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:46.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:46.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:46.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:46.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:46.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:46.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:51.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:51.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:51.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:51.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:51.736 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:51.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:51.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:51.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:52.582 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:52.584+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:38:52.582 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:52.584+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:38:56.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:38:56.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:38:56.686 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:38:56.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:38:56.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:38:56.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:38:56.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:38:56.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:38:57.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:38:57.153 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:38:57.154 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:38:57.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce60 legacy=0x7ff35000d240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:57.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:57.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350009870 legacy=0x7ff3500111f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:57.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,3,6,5,1,8] 2021-03-09T09:38:57.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:38:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163f0 addr v1:172.21.15.143:6790/0 2021-03-09T09:38:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500034a0 addr v1:172.21.15.173:6790/0 2021-03-09T09:38:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350009870 addr v1:172.21.15.143:6792/0 2021-03-09T09:38:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016220 con 0x7ff3500163f0 2021-03-09T09:38:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071e0 con 0x7ff3500034a0 2021-03-09T09:38:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500068e0 con 0x7ff350009870 2021-03-09T09:38:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff3500058b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:38:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.144+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500034a0 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:38:57.348 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.348+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff3500058b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:38:57.348 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.348+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500034a0 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:38:57.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.752+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff3500058b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:38:57.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:57.752+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500034a0 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:38:58.554 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:58.556+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff3500058b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:38:58.554 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:38:58.556+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500034a0 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:39:00.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:00.160+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff3500058b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:39:00.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:00.160+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500034a0 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:39:01.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:01.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:01.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:01.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:01.687 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:01.684+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350009870 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:39:01.688 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:01.684+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350009870 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:01.688 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:01.684+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350009870 0x7ff35001b8e0 :-1 s=OPENED pgs=169 cs=1 l=1).handle_message read tag failed 2021-03-09T09:39:01.688 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:01.684+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350009870 0x7ff35001b8e0 :-1 s=OPENED pgs=169 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:39:01.688 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:01.684+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:39:01.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:01.732+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:01.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:01.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:03.364 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:03.364+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff3500058b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:39:03.364 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:03.364+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500034a0 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:39:06.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:06.688+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:06.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:06.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:06.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:06.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:06.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:06.744+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:09.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:09.772+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff3500058b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:39:09.771 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:09.772+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500034a0 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:39:11.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:11.688+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:11.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:11.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:11.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:11.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:11.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:11.744+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:16.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:16.688+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:16.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:16.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:16.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:16.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:16.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:16.744+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:21.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:21.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:21.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:21.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:21.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:21.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:21.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:21.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:22.583 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:22.584+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff3500058b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:39:22.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:22.584+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500034a0 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:39:26.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:26.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:26.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:26.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:26.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:26.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:26.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:26.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:39:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:39:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:39:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350009870 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500034a0 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff3500058b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,8,1] 2021-03-09T09:39:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001c020 addr v1:172.21.15.173:6791/0 2021-03-09T09:39:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001c450 addr v1:172.21.15.143:6793/0 2021-03-09T09:39:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500111f0 addr v1:172.21.15.173:6789/0 2021-03-09T09:39:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500167d0 con 0x7ff3500111f0 2021-03-09T09:39:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016220 con 0x7ff35001c020 2021-03-09T09:39:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071e0 con 0x7ff35001c450 2021-03-09T09:39:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.144+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:39:27.348 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.348+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:39:27.752 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:27.752+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:39:28.553 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:28.556+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:39:30.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:30.160+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:39:31.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:31.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:31.686 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.684+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c450 legacy=0x7ff35001b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:39:31.686 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.684+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c450 legacy=0x7ff35001b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:31.687 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.684+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c450 0x7ff35001b120 :-1 s=OPENED pgs=184 cs=1 l=1).handle_message read tag failed 2021-03-09T09:39:31.687 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.684+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c450 0x7ff35001b120 :-1 s=OPENED pgs=184 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:39:31.687 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.684+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:39:31.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:31.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:31.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:31.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:31.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:31.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:31.742 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.740+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff350005fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:31.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.740+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff350005fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:31.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.740+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 0x7ff350005fc0 :-1 s=OPENED pgs=180 cs=1 l=1).handle_message read tag failed 2021-03-09T09:39:31.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.740+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 0x7ff350005fc0 :-1 s=OPENED pgs=180 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:39:31.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.740+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:39:31.744 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:31.740+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:39:33.363 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:33.364+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:39:36.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:36.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:36.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:36.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:36.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:36.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:36.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:36.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:39.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:39.772+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:39:41.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:41.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:41.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:41.684+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:41.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:41.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:41.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:41.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:46.685 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:46.684+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:46.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:46.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:46.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:46.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:46.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:46.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:51.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:51.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:51.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:51.688+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:51.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:51.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:51.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:51.744+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:52.583 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:52.584+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:39:56.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:39:56.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:39:56.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:39:56.688+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:39:56.737 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:39:56.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:39:56.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:39:56.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:39:57.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:39:57.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:39:57.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:39:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c450 legacy=0x7ff35001b120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff350005fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,7,8,1,4,5,6,2,3] 2021-03-09T09:39:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35001c020 addr v1:172.21.15.143:6789/0 2021-03-09T09:39:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001c450 addr v1:172.21.15.173:6792/0 2021-03-09T09:39:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500163f0 addr v1:172.21.15.143:6793/0 2021-03-09T09:39:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011600 con 0x7ff35001c020 2021-03-09T09:39:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500167d0 con 0x7ff35001c450 2021-03-09T09:39:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:39:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071e0 con 0x7ff3500163f0 2021-03-09T09:39:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.144+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:57.347 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.348+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:39:57.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:57.752+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:39:58.552 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:39:58.556+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:40:00.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:00.160+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:40:01.689 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:01.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:01.689 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:01.688+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:01.690 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.688+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:40:01.690 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.688+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:01.690 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.688+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163f0 0x7ff35000fa50 :-1 s=OPENED pgs=197 cs=1 l=1).handle_message read tag failed 2021-03-09T09:40:01.691 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.688+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163f0 0x7ff35000fa50 :-1 s=OPENED pgs=197 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:40:01.691 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.688+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:40:01.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:01.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:01.738 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.736+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:01.739 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.736+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:01.739 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.736+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 0x7ff35001b8e0 :-1 s=OPENED pgs=191 cs=1 l=1).handle_message read tag failed 2021-03-09T09:40:01.739 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.736+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 0x7ff35001b8e0 :-1 s=OPENED pgs=191 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:40:01.740 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:01.736+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:40:01.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:01.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:03.362 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:03.364+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:40:06.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:06.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:06.694 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:06.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:06.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:06.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:06.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:06.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:09.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:09.772+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:40:11.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:11.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:11.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:11.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:11.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:11.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:11.743 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:11.744+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:16.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:16.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:16.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:16.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:16.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:16.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:16.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:16.740+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:21.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:21.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:21.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:21.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:21.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:21.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:21.743 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:21.744+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:22.578 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:22.580+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:40:26.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:26.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:26.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:26.688+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:26.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:26.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:26.768 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:26.768+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:40:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:40:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:40:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 legacy=0x7ff35001b8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350006ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,4,3,5,2,6] 2021-03-09T09:40:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500111f0 addr v1:172.21.15.173:6789/0 2021-03-09T09:40:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500097b0 addr v1:172.21.15.143:6791/0 2021-03-09T09:40:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35000d550 addr v1:172.21.15.173:6790/0 2021-03-09T09:40:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500064a0 con 0x7ff3500111f0 2021-03-09T09:40:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011600 con 0x7ff35000d550 2021-03-09T09:40:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009b90 con 0x7ff3500097b0 2021-03-09T09:40:27.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:27.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500097b0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:40:27.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:40:27.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.148+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000d550 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:40:27.350 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.352+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500097b0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:40:27.351 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.352+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:40:27.351 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.352+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000d550 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:40:27.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.756+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500097b0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:40:27.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.756+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:40:27.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:27.756+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000d550 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:40:28.558 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:28.560+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500097b0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:40:28.559 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:28.560+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:40:28.559 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:28.560+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000d550 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:40:30.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:30.164+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500097b0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:40:30.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:30.164+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000d550 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:40:30.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:30.164+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:40:31.687 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:31.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:31.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:31.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:31.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:31.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:31.770 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:31.768+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:33.364 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:33.368+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500097b0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:40:33.364 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:33.368+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:40:33.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:33.368+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000d550 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:40:36.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:36.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:36.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:36.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:36.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:36.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:36.796 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:36.796+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:39.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:39.772+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500097b0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:40:39.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:39.772+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000d550 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:40:39.771 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:39.772+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:40:41.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:41.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:41.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:41.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:41.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:41.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:41.798 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:41.800+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:46.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:46.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:46.689 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:46.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:46.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:46.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:46.820 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:46.820+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:51.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:51.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:51.689 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:51.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:51.740 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:51.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:51.822 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:51.820+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:52.573 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:52.576+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:40:52.582 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:52.584+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000d550 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:40:52.582 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:52.584+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500097b0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:40:56.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:40:56.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:40:56.688 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:40:56.688+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:40:56.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:40:56.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:40:56.844 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:40:56.844+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:40:57.146 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:40:57.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:40:57.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:40:57.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500097b0 legacy=0x7ff35000ce40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:57.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000d550 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:40:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,2,5] 2021-03-09T09:40:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500097b0 addr v1:172.21.15.143:6792/0 2021-03-09T09:40:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163f0 addr v1:172.21.15.143:6790/0 2021-03-09T09:40:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001c020 addr v1:172.21.15.173:6791/0 2021-03-09T09:40:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010870 con 0x7ff3500163f0 2021-03-09T09:40:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500064a0 con 0x7ff35001c020 2021-03-09T09:40:57.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011600 con 0x7ff3500097b0 2021-03-09T09:40:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:40:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.148+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:57.346 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.352+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:40:57.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:57.756+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:40:58.558 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:40:58.560+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:41:00.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:00.164+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:41:01.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:01.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:01.689 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.692+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:01.689 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.692+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:01.690 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.692+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500097b0 0x7ff350006a90 :-1 s=OPENED pgs=223 cs=1 l=1).handle_message read tag failed 2021-03-09T09:41:01.690 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.692+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500097b0 0x7ff350006a90 :-1 s=OPENED pgs=223 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:41:01.690 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.692+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:41:01.690 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:01.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:01.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:01.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:01.846 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:01.844+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:01.847 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.848+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:41:01.847 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.848+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:01.847 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.848+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 0x7ff35000ce10 :-1 s=OPENED pgs=219 cs=1 l=1).handle_message read tag failed 2021-03-09T09:41:01.848 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.848+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 0x7ff35000ce10 :-1 s=OPENED pgs=219 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:41:01.848 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:01.848+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:41:03.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:03.368+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:41:06.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:06.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:06.689 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:06.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:06.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:06.736+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:06.867 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:06.868+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:09.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:09.776+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:41:11.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:11.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:11.689 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:11.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:11.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:11.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:11.870 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:11.868+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:16.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:16.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:16.689 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:16.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:16.738 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:16.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:16.872 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:16.872+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:21.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:21.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:21.689 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:21.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:21.739 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:21.744+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:21.891 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:21.892+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:22.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:22.584+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:41:26.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:26.688+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:26.689 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:26.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:26.739 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:26.740+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:26.894 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:26.892+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:27.146 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:41:27.146 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:41:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:41:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:27.147 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,3,5,4,2,8,6,0,7] 2021-03-09T09:41:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500111f0 addr v1:172.21.15.173:6789/0 2021-03-09T09:41:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35001c450 addr v1:172.21.15.173:6790/0 2021-03-09T09:41:27.148 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001c020 addr v1:172.21.15.173:6791/0 2021-03-09T09:41:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009b90 con 0x7ff3500111f0 2021-03-09T09:41:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010870 con 0x7ff35001c450 2021-03-09T09:41:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007260 con 0x7ff35001c020 2021-03-09T09:41:27.149 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:41:27.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.152+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:41:27.350 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.356+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:41:27.351 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.356+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:41:27.755 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.760+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:41:27.756 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:27.760+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:41:28.561 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:28.564+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:41:28.561 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:28.564+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:41:30.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:30.168+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:41:30.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:30.168+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:41:31.688 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:31.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:31.689 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:31.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:31.739 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:31.744+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:31.915 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:31.916+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:31.916 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:31.916+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:41:31.916 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:31.916+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:31.916 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:31.916+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 0x7ff35000ce10 :-1 s=OPENED pgs=231 cs=1 l=1).handle_message read tag failed 2021-03-09T09:41:31.916 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:31.916+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 0x7ff35000ce10 :-1 s=OPENED pgs=231 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:41:31.917 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:31.916+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:41:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:33.372+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:41:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:33.372+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:41:36.689 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:36.696+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:36.690 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:36.696+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:36.739 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:36.744+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:36.917 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:36.920+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:39.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:39.779+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:41:39.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:39.779+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:41:41.690 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:41.694+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:41.691 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:41.694+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:41.740 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:41.743+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:41.940 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:41.943+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:46.692 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:46.693+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:46.692 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:46.693+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:46.741 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:46.742+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:46.943 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:46.942+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:51.692 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:51.692+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:51.693 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:51.692+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:51.742 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:51.741+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:51.969 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:51.969+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:52.591 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:52.592+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:41:52.592 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:52.592+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:41:56.693 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:41:56.696+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:41:56.694 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:41:56.696+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:41:56.742 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:41:56.741+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:41:56.972 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:41:56.969+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:41:57.150 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:41:57.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:41:57.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:41:57.151 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:57.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c450 legacy=0x7ff35001b8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:57.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:57.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,2,6,0,7,4] 2021-03-09T09:41:57.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:41:57.152 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500170f0 addr v1:172.21.15.143:6793/0 2021-03-09T09:41:57.153 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500097b0 addr v1:172.21.15.143:6790/0 2021-03-09T09:41:57.153 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500111f0 addr v1:172.21.15.143:6792/0 2021-03-09T09:41:57.153 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011790 con 0x7ff3500097b0 2021-03-09T09:41:57.153 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009b90 con 0x7ff3500111f0 2021-03-09T09:41:57.154 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010870 con 0x7ff3500170f0 2021-03-09T09:41:57.154 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:57.154 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.152+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:41:57.352 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.356+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:41:57.756 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:57.760+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:41:58.561 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:41:58.563+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:42:00.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:00.167+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:42:01.694 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:01.695+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:01.695 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:42:01.695 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:01.695 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35000ce10 :-1 s=OPENED pgs=257 cs=1 l=1).handle_message read tag failed 2021-03-09T09:42:01.696 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35000ce10 :-1 s=OPENED pgs=257 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:42:01.696 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:42:01.696 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:01.695+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500170f0 legacy=0x7ff3500194e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500170f0 legacy=0x7ff3500194e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:01.697 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500170f0 0x7ff3500194e0 :-1 s=OPENED pgs=252 cs=1 l=1).handle_message read tag failed 2021-03-09T09:42:01.698 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500170f0 0x7ff3500194e0 :-1 s=OPENED pgs=252 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:42:01.698 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:01.695+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:42:01.743 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:01.744+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:01.994 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:01.992+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:03.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:03.371+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:42:06.695 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:06.694+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:06.695 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:06.694+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:06.744 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:06.744+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:06.997 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:06.996+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:09.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:09.774+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:42:11.695 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:11.694+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:11.696 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:11.694+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:11.744 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:11.743+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:12.019 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:12.019+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:16.696 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:16.693+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:16.696 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:16.693+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:16.745 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:16.743+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:17.022 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:17.019+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:21.696 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:21.693+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:21.697 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:21.693+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:21.745 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:21.742+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:22.043 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:22.042+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:22.590 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:22.585+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:42:26.697 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:26.697+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:26.697 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:26.697+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:26.745 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:26.742+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:27.052 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:27.050+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:27.153 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:42:27.153 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:42:27.153 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:42:27.154 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500170f0 legacy=0x7ff3500194e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:27.154 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:27.154 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500097b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:27.154 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,4,7] 2021-03-09T09:42:27.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163f0 addr v1:172.21.15.143:6789/0 2021-03-09T09:42:27.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001c020 addr v1:172.21.15.143:6791/0 2021-03-09T09:42:27.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001c450 addr v1:172.21.15.173:6792/0 2021-03-09T09:42:27.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007260 con 0x7ff3500163f0 2021-03-09T09:42:27.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011790 con 0x7ff35001c020 2021-03-09T09:42:27.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009b90 con 0x7ff35001c450 2021-03-09T09:42:27.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:27.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:42:27.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.149+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:42:27.353 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.353+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:42:27.353 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.353+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:42:27.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.757+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:42:27.758 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:27.757+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:42:28.563 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:28.560+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:42:28.563 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:28.560+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:42:30.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:30.164+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:42:30.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:30.164+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:42:31.697 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:31.696+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:31.698 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:31.696+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:31.746 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:31.746+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:31.747 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:31.744+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 legacy=0x7ff350014190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:42:31.747 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:31.744+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 legacy=0x7ff350014190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:31.747 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:31.744+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 0x7ff350014190 :-1 s=OPENED pgs=259 cs=1 l=1).handle_message read tag failed 2021-03-09T09:42:31.748 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:31.744+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 0x7ff350014190 :-1 s=OPENED pgs=259 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:42:31.748 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:31.744+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:42:32.072 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:32.070+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:33.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:33.368+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:42:33.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:33.368+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:42:36.697 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:36.696+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:36.698 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:36.696+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:36.746 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:36.742+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:37.074 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:37.070+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:39.775 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:39.776+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:42:39.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:39.776+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:42:41.698 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:41.696+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:41.698 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:41.696+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:41.747 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:41.745+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:42.096 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:42.093+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:46.698 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:46.695+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:46.699 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:46.695+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:46.747 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:46.745+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:47.099 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:47.093+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:51.699 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:51.695+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:51.700 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:51.695+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:51.748 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:51.744+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:52.121 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:52.116+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:52.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:52.579+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:42:52.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:52.579+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:42:56.699 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:42:56.694+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:42:56.700 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:42:56.694+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:42:56.749 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:42:56.743+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:42:57.124 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:42:57.119+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:42:57.154 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:42:57.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:42:57.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:42:57.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c450 legacy=0x7ff350014190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:57.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:57.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:57.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,1,5,2,8,3,0,7,4] 2021-03-09T09:42:57.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:42:57.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500111f0 addr v1:172.21.15.143:6792/0 2021-03-09T09:42:57.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:42:57.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500163f0 addr v1:172.21.15.173:6789/0 2021-03-09T09:42:57.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001c450 addr v1:172.21.15.173:6791/0 2021-03-09T09:42:57.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc10 con 0x7ff3500163f0 2021-03-09T09:42:57.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007260 con 0x7ff35001c450 2021-03-09T09:42:57.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350011790 con 0x7ff3500111f0 2021-03-09T09:42:57.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:57.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.150+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:42:57.356 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.354+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:42:57.760 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:57.758+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:42:58.564 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:42:58.562+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:43:00.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:00.166+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:43:01.699 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:01.694+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:01.700 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:01.694+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:01.701 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:01.694+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:01.701 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:01.694+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:01.701 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:01.694+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35000ce10 :-1 s=OPENED pgs=280 cs=1 l=1).handle_message read tag failed 2021-03-09T09:43:01.701 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:01.694+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35000ce10 :-1 s=OPENED pgs=280 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:43:01.702 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:01.694+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:43:01.749 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:01.747+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:02.146 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:02.143+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:02.249 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:02.142+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c450 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:43:02.250 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:02.142+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c450 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:02.250 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:02.142+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c450 0x7ff35000fa20 :-1 s=OPENED pgs=280 cs=1 l=1).handle_message read tag failed 2021-03-09T09:43:02.250 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:02.142+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c450 0x7ff35000fa20 :-1 s=OPENED pgs=280 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:43:02.251 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:02.142+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:43:03.372 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:03.370+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:43:06.700 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:06.698+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:06.700 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:06.698+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:06.750 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:06.746+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:07.149 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:07.142+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:09.775 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:09.774+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:43:11.700 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:11.698+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:11.701 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:11.698+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:11.750 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:11.746+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:12.170 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:12.166+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:16.700 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:16.697+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:16.701 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:16.697+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:16.751 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:16.746+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:17.173 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:17.166+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:21.701 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:21.697+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:21.701 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:21.697+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:21.751 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:21.745+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:22.195 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:22.189+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:22.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:22.585+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:43:26.701 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:26.697+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:26.702 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:26.697+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:26.751 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:26.745+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:27.155 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:43:27.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:43:27.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:43:27.156 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:27.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c450 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:27.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:27.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,8,7,0,2,3] 2021-03-09T09:43:27.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001c020 addr v1:172.21.15.143:6791/0 2021-03-09T09:43:27.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350009710 addr v1:172.21.15.143:6793/0 2021-03-09T09:43:27.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500158b0 addr v1:172.21.15.173:6792/0 2021-03-09T09:43:27.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001be40 con 0x7ff35001c020 2021-03-09T09:43:27.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc10 con 0x7ff3500158b0 2021-03-09T09:43:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009b20 con 0x7ff350009710 2021-03-09T09:43:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.153+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:43:27.197 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:27.189+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:27.361 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.357+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:43:27.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:27.761+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:43:28.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:28.565+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:43:30.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:30.169+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:43:31.701 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:31.697+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:31.702 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:31.697+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:31.702 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.697+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:31.703 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.697+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:31.703 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.697+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350009710 0x7ff3500151a0 :-1 s=OPENED pgs=293 cs=1 l=1).handle_message read tag failed 2021-03-09T09:43:31.703 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.697+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350009710 0x7ff3500151a0 :-1 s=OPENED pgs=293 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:43:31.703 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.697+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:43:31.756 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:31.749+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:31.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.749+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500158b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:43:31.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.749+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500158b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:31.758 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.749+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500158b0 0x7ff35001a110 :-1 s=OPENED pgs=287 cs=1 l=1).handle_message read tag failed 2021-03-09T09:43:31.758 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.749+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500158b0 0x7ff35001a110 :-1 s=OPENED pgs=287 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:43:31.758 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.749+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:43:31.758 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:31.749+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:43:32.219 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:32.213+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:33.374 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:33.373+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:43:36.702 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:36.697+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:36.703 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:36.697+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:36.752 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:36.749+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:37.222 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:37.213+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:39.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:39.776+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:43:41.702 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:41.700+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:41.702 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:41.700+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:41.752 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:41.748+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:42.244 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:42.236+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:46.702 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:46.700+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:46.702 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:46.700+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:46.752 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:46.748+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:47.246 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:47.240+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:51.703 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:51.700+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:51.703 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:51.700+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:51.753 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:51.744+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:52.268 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:52.260+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:52.594 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:52.592+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:43:56.703 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:43:56.700+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:43:56.703 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:43:56.700+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:43:56.753 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:43:56.744+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:43:57.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:43:57.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:43:57.157 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:43:57.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:57.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500158b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:57.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:57.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,3,2] 2021-03-09T09:43:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500158b0 addr v1:172.21.15.143:6789/0 2021-03-09T09:43:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350009710 addr v1:172.21.15.173:6790/0 2021-03-09T09:43:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163f0 addr v1:172.21.15.143:6790/0 2021-03-09T09:43:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007260 con 0x7ff3500158b0 2021-03-09T09:43:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:43:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001be40 con 0x7ff3500163f0 2021-03-09T09:43:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009b20 con 0x7ff350009710 2021-03-09T09:43:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:43:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.152+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:43:57.270 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:43:57.260+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:43:57.357 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.356+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:43:57.358 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.356+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:43:57.358 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.356+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:43:57.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.760+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:43:57.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.760+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:43:57.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:57.760+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:43:58.566 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:58.564+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:43:58.566 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:58.564+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:43:58.566 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:43:58.564+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:00.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:00.168+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:44:00.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:00.168+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:00.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:00.168+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:01.703 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:01.700+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:01.703 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:01.700+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:01.753 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:01.748+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:02.292 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:02.284+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:03.374 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:03.371+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:44:03.374 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:03.371+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:03.374 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:03.371+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:06.703 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:06.699+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:06.704 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:06.699+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:06.753 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:06.748+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:07.294 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:07.284+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:09.775+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:09.775+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:44:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:09.775+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:11.703 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:11.699+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:11.704 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:11.699+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:11.754 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:11.748+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:12.297 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:12.288+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:16.704 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:16.699+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:16.704 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:16.699+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:16.754 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:16.748+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:17.316 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:17.312+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:21.704 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:21.699+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:21.705 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:21.699+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:21.754 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:21.748+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:22.318 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:22.312+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:22.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:22.583+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:22.593 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:22.587+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:44:22.593 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:22.591+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:26.704 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:26.699+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:26.704 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:26.699+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:26.754 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:26.747+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:27.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:44:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:44:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:44:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,4,0,8,7,5,2,6,1] 2021-03-09T09:44:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350009710 addr v1:172.21.15.173:6790/0 2021-03-09T09:44:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001c020 addr v1:172.21.15.143:6791/0 2021-03-09T09:44:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500158b0 addr v1:172.21.15.143:6789/0 2021-03-09T09:44:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500065e0 con 0x7ff3500158b0 2021-03-09T09:44:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007260 con 0x7ff350009710 2021-03-09T09:44:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001be40 con 0x7ff35001c020 2021-03-09T09:44:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:44:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.155+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:27.340 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:27.335+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:27.362 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.359+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:27.362 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.359+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:44:27.362 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.359+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:27.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.763+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:27.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.763+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:44:27.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:27.763+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:28.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:28.567+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:28.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:28.567+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:44:28.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:28.567+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:30.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:30.171+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:30.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:30.171+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:44:30.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:30.171+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:31.704 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:31.698+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:31.705 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:31.698+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:31.754 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:31.747+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:32.342 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:32.335+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:33.378 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:33.374+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:33.378 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:33.374+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:44:33.378 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:33.374+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:36.704 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:36.698+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:36.705 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:36.698+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:36.755 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:36.751+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:37.363 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:37.359+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:39.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:39.778+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:39.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:39.778+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:44:39.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:39.778+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:41.704 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:41.698+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:41.705 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:41.698+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:41.755 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:41.747+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:42.366 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:42.359+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:46.705 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:46.702+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:46.705 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:46.702+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:46.755 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:46.751+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:47.392 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:47.387+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:51.705 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:51.702+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:51.705 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:51.702+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:51.755 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:51.751+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:52.394 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:52.387+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:52.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:52.594+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:44:52.598 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:52.594+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:44:52.598 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:52.594+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:44:56.705 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:44:56.698+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:44:56.706 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:44:56.698+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:44:56.755 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:44:56.751+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:44:57.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:44:57.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:44:57.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:44:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350009710 legacy=0x7ff3500151a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500158b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,5,1,6,2,8] 2021-03-09T09:44:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T09:44:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350015c90 addr v1:172.21.15.173:6791/0 2021-03-09T09:44:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001b830 addr v1:172.21.15.173:6789/0 2021-03-09T09:44:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009b20 con 0x7ff35001b830 2021-03-09T09:44:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500065e0 con 0x7ff350015c90 2021-03-09T09:44:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007260 con 0x7ff3500111f0 2021-03-09T09:44:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:44:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.150+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:57.358 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.354+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:44:57.415 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:44:57.411+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:44:57.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.410+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:57.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.410+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:57.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.410+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 0x7ff35001a110 :-1 s=OPENED pgs=342 cs=1 l=1).handle_message read tag failed 2021-03-09T09:44:57.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.410+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 0x7ff35001a110 :-1 s=OPENED pgs=342 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:44:57.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.410+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:44:57.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.410+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:44:57.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:57.758+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:44:58.566 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:44:58.562+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:45:00.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:00.166+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:45:01.705 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:01.698+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:01.706 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:01.698+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:01.755 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:01.751+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:01.756 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:01.750+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:01.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:01.750+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:01.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:01.750+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35000ce10 :-1 s=OPENED pgs=338 cs=1 l=1).handle_message read tag failed 2021-03-09T09:45:01.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:01.750+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35000ce10 :-1 s=OPENED pgs=338 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:45:01.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:01.750+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:45:02.418 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:02.411+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:03.374 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:03.370+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:45:06.705 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:06.702+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:06.706 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:06.702+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:06.755 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:06.751+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:07.440 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:07.435+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:09.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:09.778+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:45:11.706 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:11.701+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:11.706 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:11.701+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:11.756 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:11.751+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:12.442 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:12.435+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:16.706 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:16.701+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:16.707 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:16.701+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:16.763 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:16.751+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:17.463 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:17.459+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:21.706 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:21.701+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:21.707 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:21.701+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:21.756 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:21.750+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:22.466 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:22.458+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:22.594 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:22.589+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:45:26.706 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:26.701+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:26.707 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:26.701+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:26.757 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:26.750+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:27.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:45:27.158 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:45:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:45:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,8,6] 2021-03-09T09:45:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff350015c90 addr v1:172.21.15.143:6790/0 2021-03-09T09:45:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001c020 addr v1:172.21.15.143:6793/0 2021-03-09T09:45:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T09:45:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001be40 con 0x7ff350015c90 2021-03-09T09:45:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350009b20 con 0x7ff3500163b0 2021-03-09T09:45:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500065e0 con 0x7ff35001c020 2021-03-09T09:45:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.153+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c90 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:45:27.361 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.357+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c90 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:45:27.488 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:27.482+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:27.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:27.761+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c90 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:45:28.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:28.565+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c90 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:45:30.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:30.169+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c90 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:45:31.707 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:31.701+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:31.707 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:45:31.708 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:31.708 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350010130 :-1 s=OPENED pgs=368 cs=1 l=1).handle_message read tag failed 2021-03-09T09:45:31.708 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350010130 :-1 s=OPENED pgs=368 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:45:31.708 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:31.701+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:31.709 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:45:31.709 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:31.710 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:31.710 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 0x7ff35000fa20 :-1 s=OPENED pgs=364 cs=1 l=1).handle_message read tag failed 2021-03-09T09:45:31.710 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 0x7ff35000fa20 :-1 s=OPENED pgs=364 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:45:31.711 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:31.701+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:45:31.757 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:31.750+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:32.490 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:32.482+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:33.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:33.373+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c90 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:45:36.707 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:36.701+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:36.707 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:36.701+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:36.757 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:36.750+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:37.511 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:37.506+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:39.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:39.781+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c90 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:45:41.707 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:41.705+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:41.707 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:41.705+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:41.757 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:41.750+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:42.514 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:42.506+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:46.707 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:46.705+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:46.708 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:46.705+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:46.758 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:46.750+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:47.535 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:47.530+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:51.707 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:51.705+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:51.708 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:51.705+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:51.757 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:51.754+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:52.538 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:52.530+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:52.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:52.588+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c90 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:45:56.708 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:45:56.700+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:45:56.709 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:45:56.700+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:45:56.758 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:45:56.750+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:45:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:45:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:45:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:45:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c90 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,4,8,7,3,1,5,2,0] 2021-03-09T09:45:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:45:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T09:45:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500111f0 addr v1:172.21.15.143:6791/0 2021-03-09T09:45:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001c020 addr v1:172.21.15.143:6793/0 2021-03-09T09:45:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007230 con 0x7ff3500111f0 2021-03-09T09:45:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001be40 con 0x7ff3500163b0 2021-03-09T09:45:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500065e0 con 0x7ff35001c020 2021-03-09T09:45:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.152+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:45:57.361 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.356+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:45:57.559 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:45:57.554+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:45:57.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:57.760+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:45:58.566 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:45:58.564+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:46:00.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:00.168+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:46:01.708 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:01.704+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:01.708 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:01.704+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:01.709 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:01.710 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:01.710 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350010130 :-1 s=OPENED pgs=391 cs=1 l=1).handle_message read tag failed 2021-03-09T09:46:01.710 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350010130 :-1 s=OPENED pgs=391 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:46:01.710 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:46:01.711 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:01.711 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:46:01.711 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 0x7ff35000fa20 :-1 s=OPENED pgs=385 cs=1 l=1).handle_message read tag failed 2021-03-09T09:46:01.712 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 0x7ff35000fa20 :-1 s=OPENED pgs=385 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:46:01.712 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:01.704+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:46:01.758 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:01.754+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:02.562 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:02.554+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:03.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:03.372+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:46:06.708 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:06.700+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:06.708 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:06.700+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:06.758 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:06.750+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:07.584 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:07.578+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:09.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:09.776+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:46:11.708 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:11.704+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:11.709 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:11.704+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:11.758 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:11.749+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:12.586 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:12.577+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:16.708 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:16.700+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:16.709 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:16.700+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:16.759 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:16.753+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:17.607 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:17.601+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:21.708 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:21.704+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:21.709 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:21.704+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:21.759 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:21.753+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:22.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:22.592+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:46:22.610 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:22.601+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:26.709 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:26.704+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:26.709 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:26.704+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:26.759 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:26.753+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:46:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:46:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:46:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,2,5,1,7,0] 2021-03-09T09:46:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35000a6f0 addr v1:172.21.15.173:6790/0 2021-03-09T09:46:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff350013160 addr v1:172.21.15.143:6790/0 2021-03-09T09:46:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350015c90 addr v1:172.21.15.173:6791/0 2021-03-09T09:46:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001c530 con 0x7ff350013160 2021-03-09T09:46:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007230 con 0x7ff35000a6f0 2021-03-09T09:46:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500065e0 con 0x7ff350015c90 2021-03-09T09:46:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350013160 legacy=0x7ff350015580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:46:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.156+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:46:27.361 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.360+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350013160 legacy=0x7ff350015580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:46:27.362 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.360+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:46:27.631 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:27.625+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:27.632 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.628+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:46:27.632 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.628+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:27.633 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.628+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 0x7ff35001a110 :-1 s=OPENED pgs=406 cs=1 l=1).handle_message read tag failed 2021-03-09T09:46:27.633 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.628+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 0x7ff35001a110 :-1 s=OPENED pgs=406 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:46:27.633 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.628+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:46:27.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.764+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350013160 legacy=0x7ff350015580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:46:27.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:27.764+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:46:28.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:28.568+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350013160 legacy=0x7ff350015580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:46:28.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:28.568+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:46:30.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:30.172+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350013160 legacy=0x7ff350015580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:46:30.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:30.172+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:46:31.709 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:31.704+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:31.710 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:31.704+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:31.759 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:31.753+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:32.634 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:32.625+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:33.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:33.376+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350013160 legacy=0x7ff350015580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:46:33.378 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:33.376+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:46:36.709 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:36.703+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:36.709 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:36.703+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:36.760 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:36.753+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:37.655 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:37.649+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:39.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:39.779+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350013160 legacy=0x7ff350015580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:46:39.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:39.779+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:46:41.709 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:41.703+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:41.710 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:41.703+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:41.759 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:41.753+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:42.658 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:42.649+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:46.709 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:46.707+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:46.710 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:46.707+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:46.760 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:46.757+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:47.679 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:47.673+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:51.710 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:51.707+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:51.710 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:51.707+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:51.760 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:51.753+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:52.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:52.595+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350013160 legacy=0x7ff350015580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:46:52.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:52.595+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:46:52.681 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:52.673+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:56.710 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:46:56.707+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:46:56.710 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:46:56.707+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:46:56.760 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:46:56.753+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:46:57.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:46:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:46:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:46:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015c90 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350013160 legacy=0x7ff350015580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,0,7] 2021-03-09T09:46:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:46:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001b830 addr v1:172.21.15.173:6789/0 2021-03-09T09:46:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:46:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T09:46:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350015c90 addr v1:172.21.15.173:6792/0 2021-03-09T09:46:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001be00 con 0x7ff3500111f0 2021-03-09T09:46:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001c530 con 0x7ff35001b830 2021-03-09T09:46:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007230 con 0x7ff350015c90 2021-03-09T09:46:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:46:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.155+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:46:57.361 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.359+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:46:57.362 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.359+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:46:57.703 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:46:57.697+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:46:57.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.763+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:46:57.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:57.763+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:46:58.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:58.567+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:46:58.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:46:58.567+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:47:00.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:00.171+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:47:00.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:00.171+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:47:01.710 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:01.707+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:01.711 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:01.707+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:01.760 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:01.757+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:01.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:01.755+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:01.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:01.755+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:01.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:01.755+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 0x7ff35000ce10 :-1 s=OPENED pgs=424 cs=1 l=1).handle_message read tag failed 2021-03-09T09:47:01.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:01.755+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 0x7ff35000ce10 :-1 s=OPENED pgs=424 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:47:01.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:01.755+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:47:02.706 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:02.697+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:03.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:03.375+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:47:03.378 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:03.375+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:47:06.710 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:06.707+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:06.711 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:06.707+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:06.761 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:06.752+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:07.708 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:07.700+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:09.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:09.779+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:47:09.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:09.779+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:47:11.710 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:11.707+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:11.711 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:11.707+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:11.761 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:11.756+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:12.727 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:12.720+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:16.710 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:16.706+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:16.711 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:16.706+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:16.761 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:16.752+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:17.730 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:17.724+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:21.711 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:21.706+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:21.711 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:21.706+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:21.761 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:21.756+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:22.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:22.594+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:47:22.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:22.594+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:47:22.751 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:22.744+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:26.711 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:26.706+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:26.711 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:26.706+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:26.761 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:26.752+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:27.159 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:47:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:47:27.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:47:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001b830 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,5,7,0,6,2,1,8,3] 2021-03-09T09:47:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500163b0 addr v1:172.21.15.143:6791/0 2021-03-09T09:47:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001c020 addr v1:172.21.15.173:6791/0 2021-03-09T09:47:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350015c90 addr v1:172.21.15.173:6792/0 2021-03-09T09:47:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500065e0 con 0x7ff3500163b0 2021-03-09T09:47:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001be00 con 0x7ff35001c020 2021-03-09T09:47:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350015170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:47:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001c530 con 0x7ff350015c90 2021-03-09T09:47:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.154+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:27.360 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.358+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350015170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:47:27.753 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:27.744+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:27.755 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.750+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:27.755 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.750+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:27.755 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.750+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 0x7ff35000fa20 :-1 s=OPENED pgs=450 cs=1 l=1).handle_message read tag failed 2021-03-09T09:47:27.755 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.750+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 0x7ff35000fa20 :-1 s=OPENED pgs=450 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:47:27.756 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.750+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:47:27.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:27.762+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350015170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:47:28.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:28.566+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350015170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:47:30.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:30.170+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350015170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:47:31.711 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:31.706+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:31.712 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:31.706+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:31.761 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:31.752+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:31.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:31.758+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:47:31.763 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:31.758+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:31.763 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:31.758+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 0x7ff350010130 :-1 s=OPENED pgs=447 cs=1 l=1).handle_message read tag failed 2021-03-09T09:47:31.763 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:31.758+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 0x7ff350010130 :-1 s=OPENED pgs=447 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:47:31.763 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:31.758+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:47:32.775 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:32.768+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:33.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:33.374+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350015170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:47:36.711 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:36.706+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:36.712 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:36.706+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:36.762 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:36.756+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:37.777 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:37.772+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:39.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:39.782+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350015170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:47:41.711 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:41.706+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:41.712 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:41.706+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:41.762 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:41.756+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:42.799 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:42.792+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:46.712 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:46.710+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:46.712 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:46.710+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:46.762 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:46.756+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:47.801 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:47.792+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:51.712 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:51.710+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:51.712 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:51.710+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:51.762 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:51.756+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:52.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:52.594+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350015170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:47:52.823 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:52.820+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:56.712 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:47:56.706+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:47:56.713 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:47:56.706+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:47:56.762 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:47:56.756+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:47:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:47:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:47:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:47:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015c90 legacy=0x7ff350010130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350015170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,1,6,2,8,3] 2021-03-09T09:47:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:47:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T09:47:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500163b0 addr v1:172.21.15.173:6789/0 2021-03-09T09:47:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001c020 addr v1:172.21.15.143:6792/0 2021-03-09T09:47:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:47:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007210 con 0x7ff3500111f0 2021-03-09T09:47:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500065e0 con 0x7ff3500163b0 2021-03-09T09:47:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001c530 con 0x7ff35001c020 2021-03-09T09:47:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.154+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:47:57.361 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.358+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:47:57.361 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.358+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:47:57.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.761+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:47:57.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:57.761+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:47:57.826 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:47:57.820+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:47:58.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:58.565+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:47:58.617 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:47:58.565+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:48:00.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:00.169+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:48:00.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:00.169+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:48:01.713 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:01.709+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:01.713 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:01.709+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:01.714 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:01.709+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:48:01.714 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:01.709+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:01.714 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:01.709+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001c020 0x7ff35000fa20 :-1 s=OPENED pgs=473 cs=1 l=1).handle_message read tag failed 2021-03-09T09:48:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:01.709+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001c020 0x7ff35000fa20 :-1 s=OPENED pgs=473 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:48:01.715 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:01.709+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:48:01.762 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:01.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:02.847 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:02.843+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:03.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:03.373+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:48:03.378 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:03.373+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:48:06.712 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:06.705+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:06.713 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:06.705+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:06.763 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:06.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:07.849 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:07.843+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:09.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:09.777+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:48:09.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:09.777+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:48:11.713 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:11.709+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:11.713 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:11.709+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:11.763 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:11.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:12.871 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:12.867+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:16.713 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:16.709+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:16.713 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:16.709+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:16.763 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:16.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:17.873 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:17.867+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:21.713 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:21.709+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:21.714 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:21.709+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:21.763 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:21.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:22.594 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:22.589+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:48:22.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:22.589+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:48:22.897 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:22.891+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:26.713 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:26.709+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:26.714 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:26.709+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:26.763 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:26.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:48:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:48:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:48:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001c020 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,3,2] 2021-03-09T09:48:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35000a6f0 addr v1:172.21.15.143:6793/0 2021-03-09T09:48:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350015170 addr v1:172.21.15.173:6790/0 2021-03-09T09:48:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff350015c60 addr v1:172.21.15.143:6790/0 2021-03-09T09:48:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bda0 con 0x7ff350015c60 2021-03-09T09:48:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007210 con 0x7ff350015170 2021-03-09T09:48:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500065e0 con 0x7ff35000a6f0 2021-03-09T09:48:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c60 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.157+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff350015550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:48:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.361+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c60 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.361+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff350015550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:48:27.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.765+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c60 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:27.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:27.765+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff350015550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:48:27.897 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:27.891+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:28.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:28.569+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c60 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:28.571 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:28.569+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff350015550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:48:30.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:30.173+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c60 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:30.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:30.173+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff350015550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:48:31.713 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:31.709+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:31.714 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:31.709+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:31.715 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:31.709+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:31.715 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:31.709+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:31.715 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:31.709+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 0x7ff350010130 :-1 s=OPENED pgs=479 cs=1 l=1).handle_message read tag failed 2021-03-09T09:48:31.715 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:31.709+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 0x7ff350010130 :-1 s=OPENED pgs=479 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:48:31.716 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:31.709+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:48:31.763 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:31.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:32.919 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:32.915+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:33.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:33.377+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c60 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:33.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:33.377+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff350015550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:48:36.713 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:36.709+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:36.714 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:36.709+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:36.764 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:36.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:37.922 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:37.915+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:39.786 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:39.780+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c60 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:39.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:39.780+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff350015550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:48:41.714 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:41.712+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:41.714 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:41.712+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:41.764 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:41.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:42.947 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:42.943+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:46.714 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:46.712+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:46.714 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:46.712+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:46.764 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:46.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:47.950 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:47.943+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:51.714 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:51.708+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:51.715 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:51.708+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:51.764 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:51.759+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:52.599 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:52.596+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c60 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:52.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:52.596+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff350015550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:48:52.970 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:52.966+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:56.714 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:48:56.712+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:48:56.715 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:48:56.712+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:48:56.765 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:48:56.758+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:48:57.160 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:48:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:48:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:48:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff350015550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015c60 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:48:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,0,2,8,7,3,5,1,6] 2021-03-09T09:48:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:48:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500111f0 addr v1:172.21.15.143:6791/0 2021-03-09T09:48:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T09:48:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001c020 addr v1:172.21.15.143:6790/0 2021-03-09T09:48:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001c530 con 0x7ff3500163b0 2021-03-09T09:48:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bda0 con 0x7ff35001c020 2021-03-09T09:48:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007210 con 0x7ff3500111f0 2021-03-09T09:48:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:48:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:48:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.156+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:57.366 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.360+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:48:57.366 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.360+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:48:57.367 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.360+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:57.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.764+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:48:57.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.764+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:57.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:57.764+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:48:57.973 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:48:57.966+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:48:58.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:58.568+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:48:58.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:58.568+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:48:58.571 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:48:58.568+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:49:00.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:00.172+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:49:00.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:00.172+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:49:00.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:00.172+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:49:01.714 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:01.712+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:01.715 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:01.712+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:01.765 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:01.758+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:02.995 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:02.990+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:03.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:03.376+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:49:03.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:03.376+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:49:03.382 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:03.376+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:49:06.715 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:06.712+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:06.715 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:06.712+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:06.765 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:06.762+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:07.997 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:07.990+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:09.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:09.784+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:49:09.983 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:09.784+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:49:09.984 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:09.784+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:49:11.715 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:11.712+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:11.715 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:11.712+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:11.765 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:11.762+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:13.019 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:13.014+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:16.715 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:16.712+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:16.716 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:16.712+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:16.765 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:16.762+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:18.021 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:18.014+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:21.715 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:21.711+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:21.716 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:21.711+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:21.765 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:21.758+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:22.601 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:22.595+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:49:22.601 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:22.595+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:49:22.601 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:22.595+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:49:23.043 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:23.038+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:26.715 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:26.711+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:26.716 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:26.711+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:26.765 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:26.758+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:49:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:49:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:49:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001c020 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,7,1,8,5,6] 2021-03-09T09:49:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35000a6f0 addr v1:172.21.15.173:6790/0 2021-03-09T09:49:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350015170 addr v1:172.21.15.173:6792/0 2021-03-09T09:49:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350015580 addr v1:172.21.15.173:6789/0 2021-03-09T09:49:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350015580 2021-03-09T09:49:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001c530 con 0x7ff35000a6f0 2021-03-09T09:49:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bda0 con 0x7ff350015170 2021-03-09T09:49:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:49:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.155+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:49:27.361 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.359+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:49:27.361 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.359+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:49:27.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.763+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:49:27.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:27.763+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:49:28.045 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:28.038+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:28.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:28.567+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:49:28.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:28.567+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:49:30.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:30.171+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:49:30.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:30.171+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:49:31.715 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:31.711+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:31.716 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:31.711+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:31.766 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:31.762+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:31.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:31.763+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015170 legacy=0x7ff35000ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:31.767 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:31.763+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015170 legacy=0x7ff35000ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:31.767 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:31.763+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015170 0x7ff35000ea20 :-1 s=OPENED pgs=513 cs=1 l=1).handle_message read tag failed 2021-03-09T09:49:31.767 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:31.763+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015170 0x7ff35000ea20 :-1 s=OPENED pgs=513 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:49:31.767 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:31.763+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:49:33.071 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:33.066+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:33.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:33.375+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:49:33.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:33.375+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:49:36.716 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:36.711+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:36.716 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:36.711+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:36.766 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:36.762+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:38.073 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:38.066+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:39.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:39.779+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:49:39.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:39.779+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:49:41.716 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:41.711+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:41.716 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:41.711+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:41.766 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:41.762+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:43.095 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:43.090+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:46.716 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:46.715+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:46.716 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:46.715+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:46.766 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:46.761+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:48.097 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:48.089+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:51.716 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:51.711+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:51.717 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:51.711+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:51.766 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:51.761+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:52.594 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:52.591+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:49:52.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:52.595+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:49:53.119 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:53.113+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:49:56.716 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:49:56.711+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:49:56.717 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:49:56.711+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:49:56.766 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:49:56.761+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:49:57.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.155+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:49:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.155+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:49:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.155+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:49:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350015170 legacy=0x7ff35000ea20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cdf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.155+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,8,6] 2021-03-09T09:49:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:49:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001b110 addr v1:172.21.15.173:6791/0 2021-03-09T09:49:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500111f0 addr v1:172.21.15.143:6793/0 2021-03-09T09:49:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350015580 addr v1:172.21.15.143:6792/0 2021-03-09T09:49:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007210 con 0x7ff35001b110 2021-03-09T09:49:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350015580 2021-03-09T09:49:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001c530 con 0x7ff3500111f0 2021-03-09T09:49:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff372ffd700 0 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001b110 legacy=0x7ff35001b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T09:49:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001b110 legacy=0x7ff35001b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2021-03-09T09:49:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001b110 0x7ff35001b4f0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2021-03-09T09:49:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:49:57.159+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:49:58.121 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:49:58.113+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:01.717 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:01.715+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:01.717 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:01.715+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:01.718 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:01.719 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:01.719 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:01.719 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a90 :-1 s=OPENED pgs=524 cs=1 l=1).handle_message read tag failed 2021-03-09T09:50:01.719 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:01.719 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a90 :-1 s=OPENED pgs=524 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:50:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 0x7ff35000ce10 :-1 s=OPENED pgs=535 cs=1 l=1).handle_message read tag failed 2021-03-09T09:50:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 0x7ff35000ce10 :-1 s=OPENED pgs=535 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:50:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:50:01.720 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:01.715+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:50:01.767 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:01.761+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:03.143 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:03.137+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:06.717 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:06.714+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:06.718 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:06.714+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:06.767 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:06.761+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:08.163 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:08.157+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:11.717 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:11.714+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:11.718 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:11.714+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:11.767 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:11.761+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:13.165 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:13.157+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:16.717 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:16.714+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:16.718 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:16.714+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:16.767 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:16.761+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:18.187 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:18.181+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:21.717 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:21.714+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:21.718 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:21.714+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:21.767 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:21.761+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:23.189 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:23.181+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:26.718 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:26.714+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:26.718 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:26.714+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:26.767 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:26.761+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:27.161 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:50:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:50:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:50:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:27.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001b110 legacy=0x7ff35001b4f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,6,8,4,3,0,2,1,5] 2021-03-09T09:50:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500163b0 addr v1:172.21.15.173:6792/0 2021-03-09T09:50:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350015580 addr v1:172.21.15.143:6792/0 2021-03-09T09:50:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500111f0 addr v1:172.21.15.143:6793/0 2021-03-09T09:50:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d6d0 con 0x7ff350015580 2021-03-09T09:50:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500163b0 2021-03-09T09:50:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d540 con 0x7ff3500111f0 2021-03-09T09:50:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:27.158+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:28.211 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:28.205+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:31.718 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:31.714+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:31.719 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:31.719 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:31.719 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:31.719 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:31.714+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:31.720 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:31.720 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 0x7ff35000ce10 :-1 s=OPENED pgs=549 cs=1 l=1).handle_message read tag failed 2021-03-09T09:50:31.721 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a90 :-1 s=OPENED pgs=538 cs=1 l=1).handle_message read tag failed 2021-03-09T09:50:31.721 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 0x7ff35000ce10 :-1 s=OPENED pgs=549 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:50:31.721 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a90 :-1 s=OPENED pgs=538 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:50:31.721 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:50:31.722 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.714+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:50:31.768 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:31.765+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:31.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.766+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:31.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.766+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:31.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.766+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff35000fa20 :-1 s=OPENED pgs=539 cs=1 l=1).handle_message read tag failed 2021-03-09T09:50:31.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.766+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff35000fa20 :-1 s=OPENED pgs=539 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:50:31.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:31.766+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:50:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:32.294+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 legacy=0x7ff354032750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000194) for more than 900000000 us, fault. 2021-03-09T09:50:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:32.294+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 0x7ff354032750 :-1 s=OPENED pgs=30 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:50:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:32.294+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:50:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:32.294+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 legacy=0x7ff354032750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:32.294+0000 7ff3737fe700 10 monclient(hunting): get_auth_request con 0x7ff354043c70 auth_method 0 2021-03-09T09:50:33.213 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:33.205+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:36.718 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:36.714+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:36.719 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:36.714+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:36.768 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:36.765+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:38.234 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:38.229+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:41.718 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:41.714+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:41.719 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:41.714+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:41.768 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:41.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:43.236 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:43.228+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:46.718 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:46.717+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:46.719 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:46.717+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:46.768 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:46.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:48.259 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:48.252+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:51.718 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:51.713+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:51.719 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:51.713+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:51.768 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:51.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:53.261 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:53.256+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:56.719 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:50:56.717+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:50:56.719 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:50:56.717+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:50:56.768 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:50:56.760+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:50:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:50:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:50:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:50:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,4,3,2,5,0] 2021-03-09T09:50:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350015580 addr v1:172.21.15.173:6789/0 2021-03-09T09:50:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500163b0 addr v1:172.21.15.143:6791/0 2021-03-09T09:50:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T09:50:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007210 con 0x7ff350015580 2021-03-09T09:50:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500111f0 2021-03-09T09:50:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bda0 con 0x7ff3500163b0 2021-03-09T09:50:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:50:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:50:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.157+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:50:57.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.361+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:50:57.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.361+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:50:57.366 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.361+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:50:57.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.765+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:50:57.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.765+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:50:57.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:57.765+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:50:58.282 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:50:58.276+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:50:58.573 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:58.569+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:50:58.573 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:58.569+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:50:58.573 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:50:58.569+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:51:00.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:00.173+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:51:00.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:00.173+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:51:00.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:00.173+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:51:01.719 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:01.713+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:01.719 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:01.713+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:01.769 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:01.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:03.285 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:03.276+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:03.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:03.377+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:51:03.382 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:03.377+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:51:03.382 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:03.377+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:51:06.719 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:06.717+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:06.720 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:06.717+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:06.769 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:06.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:08.307 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:08.300+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:09.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:09.785+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:51:09.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:09.785+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:51:09.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:09.785+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:51:11.719 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:11.717+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:11.720 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:11.717+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:11.769 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:11.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:13.308 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:13.304+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:16.719 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:16.717+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:16.720 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:16.717+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:16.769 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:16.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:18.335 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:18.328+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:21.719 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:21.717+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:21.720 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:21.717+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:21.769 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:21.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:22.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:22.593+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:51:22.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:22.593+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:51:22.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:22.597+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:51:23.337 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:23.332+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:26.720 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:26.716+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:26.720 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:26.716+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:26.769 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:26.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:51:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:51:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:51:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,5,2] 2021-03-09T09:51:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35001c020 addr v1:172.21.15.143:6789/0 2021-03-09T09:51:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000a6f0 addr v1:172.21.15.173:6791/0 2021-03-09T09:51:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff350015170 addr v1:172.21.15.143:6790/0 2021-03-09T09:51:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015960 con 0x7ff35001c020 2021-03-09T09:51:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007210 con 0x7ff350015170 2021-03-09T09:51:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff35000a6f0 2021-03-09T09:51:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:51:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.156+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:27.362 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.360+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:51:27.363 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.360+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:27.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.764+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:27.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:27.764+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:51:28.358 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:28.356+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:28.359 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:28.352+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:28.360 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:28.352+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:28.360 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:28.352+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 0x7ff3500170f0 :-1 s=OPENED pgs=568 cs=1 l=1).handle_message read tag failed 2021-03-09T09:51:28.360 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:28.352+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 0x7ff3500170f0 :-1 s=OPENED pgs=568 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:51:28.360 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:28.352+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:51:28.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:28.568+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:28.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:28.568+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:51:30.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:30.172+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:30.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:30.172+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:51:31.721 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:31.716+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:31.723 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:31.716+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:31.770 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:31.764+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:33.361 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:33.355+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:33.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:33.376+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:33.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:33.376+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:51:36.720 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:36.716+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:36.721 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:36.716+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:36.770 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:36.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:38.382 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:38.379+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:39.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:39.784+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:39.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:39.784+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:51:41.720 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:41.716+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:41.721 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:41.716+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:41.770 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:41.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:43.385 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:43.379+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:46.720 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:46.716+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:46.721 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:46.716+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:46.770 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:46.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:48.406 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:48.403+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:51.724 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:51.716+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:51.725 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:51.716+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:51.770 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:51.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:52.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:52.596+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:52.601 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:52.596+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:51:53.409 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:53.403+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:56.721 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:51:56.720+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:51:56.721 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:51:56.720+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:51:56.770 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:51:56.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:51:57.162 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:51:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:51:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:51:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001c020 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:51:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:51:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,3,7,0,8,5,1,6,4] 2021-03-09T09:51:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T09:51:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350015580 addr v1:172.21.15.173:6790/0 2021-03-09T09:51:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500163b0 addr v1:172.21.15.173:6792/0 2021-03-09T09:51:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bda0 con 0x7ff3500111f0 2021-03-09T09:51:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015960 con 0x7ff350015580 2021-03-09T09:51:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007210 con 0x7ff3500163b0 2021-03-09T09:51:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:51:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.160+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:57.364 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.364+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:57.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.364+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:51:57.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.768+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:57.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:57.768+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:51:58.430 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:51:58.427+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:51:58.572 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:58.572+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:51:58.573 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:51:58.572+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:52:00.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:00.176+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:52:00.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:00.176+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:52:01.721 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:01.720+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:01.721 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:01.720+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:01.771 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:01.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:01.771 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:01.768+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:52:01.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:01.768+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:01.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:01.768+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff35000fa20 :-1 s=OPENED pgs=580 cs=1 l=1).handle_message read tag failed 2021-03-09T09:52:01.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:01.768+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff35000fa20 :-1 s=OPENED pgs=580 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:52:01.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:01.768+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:52:03.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:03.380+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:52:03.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:03.380+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:52:03.433 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:03.427+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:06.721 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:06.720+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:06.722 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:06.720+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:06.771 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:06.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:08.454 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:08.451+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:09.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:09.783+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:52:09.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:09.783+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:52:11.721 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:11.719+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:11.722 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:11.719+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:11.771 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:11.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:13.457 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:13.451+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:16.721 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:16.719+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:16.722 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:16.719+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:16.771 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:16.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:18.478 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:18.475+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:21.721 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:21.719+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:21.722 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:21.719+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:21.771 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:21.767+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:22.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:22.599+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:52:22.601 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:22.599+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:52:23.481 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:23.475+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:26.722 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:26.719+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:26.722 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:26.719+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:26.771 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:26.766+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:52:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:52:27.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:52:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,5,8,1,6,4] 2021-03-09T09:52:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35000a6f0 addr v1:172.21.15.143:6789/0 2021-03-09T09:52:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350013160 addr v1:172.21.15.173:6791/0 2021-03-09T09:52:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350018810 addr v1:172.21.15.143:6793/0 2021-03-09T09:52:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff35000a6f0 2021-03-09T09:52:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bda0 con 0x7ff350013160 2021-03-09T09:52:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015960 con 0x7ff350018810 2021-03-09T09:52:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.159+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:52:27.364 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.363+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:52:27.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:27.767+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:52:28.502 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:28.498+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:28.503 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:28.499+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:28.503 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:28.499+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:28.504 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:28.499+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013160 0x7ff3500170f0 :-1 s=OPENED pgs=596 cs=1 l=1).handle_message read tag failed 2021-03-09T09:52:28.504 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:28.499+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013160 0x7ff3500170f0 :-1 s=OPENED pgs=596 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:52:28.504 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:28.499+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:52:28.572 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:28.571+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:52:30.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:30.175+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:52:31.722 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:31.719+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:31.723 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:31.719+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350018810 legacy=0x7ff35001ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:52:31.723 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:31.719+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350018810 legacy=0x7ff35001ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:31.723 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:31.719+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350018810 0x7ff35001ac50 :-1 s=OPENED pgs=592 cs=1 l=1).handle_message read tag failed 2021-03-09T09:52:31.723 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:31.719+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350018810 0x7ff35001ac50 :-1 s=OPENED pgs=592 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:52:31.724 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:31.719+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:31.724 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:31.719+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:52:31.724 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:31.719+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:52:31.771 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:31.766+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:33.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:33.379+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:52:33.505 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:33.498+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:36.722 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:36.719+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:36.723 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:36.719+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:36.772 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:36.766+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:38.526 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:38.522+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:39.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:39.783+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:52:41.722 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:41.719+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:41.723 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:41.719+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:41.772 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:41.766+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:43.528 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:43.522+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:46.722 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:46.719+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:46.723 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:46.719+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:46.772 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:46.766+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:48.550 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:48.546+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:51.723 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:51.718+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:51.723 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:51.718+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:51.772 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:51.766+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:52.598 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:52.594+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:52:53.553 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:53.546+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:52:56.723 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:52:56.718+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:52:56.723 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:52:56.718+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:52:56.772 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:52:56.766+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:52:57.163 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:52:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:52:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:52:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350018810 legacy=0x7ff35001ac50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,4,6] 2021-03-09T09:52:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350013160 addr v1:172.21.15.173:6789/0 2021-03-09T09:52:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350018810 addr v1:172.21.15.143:6791/0 2021-03-09T09:52:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500111f0 addr v1:172.21.15.143:6792/0 2021-03-09T09:52:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000ab70 con 0x7ff350013160 2021-03-09T09:52:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350018810 2021-03-09T09:52:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bda0 con 0x7ff3500111f0 2021-03-09T09:52:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350018810 legacy=0x7ff3500033d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:52:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.158+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013160 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:52:57.364 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.362+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350018810 legacy=0x7ff3500033d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:52:57.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.362+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013160 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:52:57.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.766+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350018810 legacy=0x7ff3500033d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:52:57.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:57.766+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013160 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:52:58.572 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:58.570+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350018810 legacy=0x7ff3500033d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:52:58.573 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:52:58.570+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013160 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:52:58.574 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:52:58.570+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:00.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:00.174+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350018810 legacy=0x7ff3500033d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:53:00.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:00.174+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013160 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:53:01.723 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:01.722+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:01.723 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:01.722+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:53:01.724 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:01.722+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:01.725 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:01.722+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:01.725 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:01.722+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35000fa20 :-1 s=OPENED pgs=617 cs=1 l=1).handle_message read tag failed 2021-03-09T09:53:01.725 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:01.722+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35000fa20 :-1 s=OPENED pgs=617 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:53:01.725 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:01.722+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:53:01.772 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:01.770+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:03.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:03.378+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350018810 legacy=0x7ff3500033d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:53:03.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:03.378+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013160 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:53:03.577 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:03.570+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:06.723 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:06.718+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:06.723 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:06.718+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:06.773 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:06.770+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:08.579 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:08.574+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:09.786 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:09.782+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350018810 legacy=0x7ff3500033d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:53:09.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:09.782+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013160 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:53:11.723 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:11.718+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:11.724 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:11.718+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:11.773 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:11.770+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:13.598 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:13.594+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:16.723 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:16.718+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:16.724 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:16.718+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:16.773 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:16.770+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:18.601 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:18.594+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:21.724 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:21.722+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:21.724 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:21.722+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:21.773 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:21.766+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:22.599 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:22.598+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350018810 legacy=0x7ff3500033d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:53:22.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:22.598+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013160 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:53:23.622 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:23.618+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:26.724 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:26.718+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:26.724 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:26.718+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:26.773 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:26.770+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:53:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:53:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:53:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350018810 legacy=0x7ff3500033d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013160 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,0,3,8,7,4,6,1,5] 2021-03-09T09:53:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T09:53:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35000a6f0 addr v1:172.21.15.143:6789/0 2021-03-09T09:53:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350015580 addr v1:172.21.15.173:6790/0 2021-03-09T09:53:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff35000a6f0 2021-03-09T09:53:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000ab70 con 0x7ff3500163b0 2021-03-09T09:53:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350015580 2021-03-09T09:53:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:53:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350018100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:53:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.158+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:53:27.364 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.362+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:53:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.362+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350018100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:53:27.365 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.362+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:53:27.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.766+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:53:27.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.766+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350018100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:53:27.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:27.766+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:53:28.572 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:28.570+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350018100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:53:28.573 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:28.570+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:53:28.573 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:28.570+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:53:28.624 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:28.618+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:30.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:30.174+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350018100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:53:30.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:30.174+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:53:30.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:30.174+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:53:31.724 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:31.717+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:31.725 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:31.721+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:31.773 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:31.770+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:33.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:33.377+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:53:33.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:33.377+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350018100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:53:33.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:33.377+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:53:33.646 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:33.641+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:36.724 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:36.721+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:36.725 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:36.721+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:36.773 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:36.769+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:38.648 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:38.641+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:39.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:39.785+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:53:39.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:39.785+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350018100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:53:39.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:39.785+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:53:41.724 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:41.721+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:41.725 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:41.721+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:41.774 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:41.769+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:43.670 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:43.665+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:46.725 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:46.721+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:46.725 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:46.721+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:46.774 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:46.769+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:48.673 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:48.665+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:51.725 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:51.721+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:51.725 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:51.721+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:51.774 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:51.769+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:52.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:52.597+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350018100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:53:52.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:52.597+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:53:52.601 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:52.597+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:53:53.694 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:53.689+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:53:56.725 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:53:56.721+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:53:56.726 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:53:56.721+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:53:56.774 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:53:56.769+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:53:57.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:53:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:53:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:53:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015580 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350018100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:53:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,6,5,7,8,1] 2021-03-09T09:53:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:53:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500111f0 addr v1:172.21.15.143:6791/0 2021-03-09T09:53:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350013160 addr v1:172.21.15.143:6792/0 2021-03-09T09:53:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500163b0 addr v1:172.21.15.173:6791/0 2021-03-09T09:53:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bda0 con 0x7ff3500111f0 2021-03-09T09:53:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:53:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500163b0 2021-03-09T09:53:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000ab70 con 0x7ff350013160 2021-03-09T09:53:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff3737fe700 0 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T09:53:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2021-03-09T09:53:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2021-03-09T09:53:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.157+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:53:57.364 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.361+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:53:57.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:57.765+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:53:58.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:53:58.569+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:53:58.696 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:53:58.689+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:00.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:00.173+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:54:01.725 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:01.725+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:01.726 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:01.725+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:01.727 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:01.725+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350013160 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:54:01.727 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:01.725+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350013160 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:01.727 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:01.725+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350013160 0x7ff35001c4f0 :-1 s=OPENED pgs=650 cs=1 l=1).handle_message read tag failed 2021-03-09T09:54:01.727 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:01.725+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350013160 0x7ff35001c4f0 :-1 s=OPENED pgs=650 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:54:01.728 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:01.725+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:54:01.774 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:01.769+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:03.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:03.377+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:54:03.718 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:03.713+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:06.725 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:06.724+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:06.726 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:06.724+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:06.774 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:06.773+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:08.720 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:08.713+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:09.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:09.780+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:54:11.726 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:11.724+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:11.727 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:11.724+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:11.774 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:11.773+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:13.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:13.737+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:16.726 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:16.724+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:16.727 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:16.724+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:16.776 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:16.773+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:18.744 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:18.737+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:21.726 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:21.720+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:21.727 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:21.720+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:21.775 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:21.773+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:22.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:22.596+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:54:23.766 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:23.761+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:26.726 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:26.724+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:26.727 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:26.724+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:26.775 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:26.769+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:27.164 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:54:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:54:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:54:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350013160 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,8,1] 2021-03-09T09:54:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35000a6f0 addr v1:172.21.15.173:6792/0 2021-03-09T09:54:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350015580 addr v1:172.21.15.143:6793/0 2021-03-09T09:54:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350010140 addr v1:172.21.15.173:6789/0 2021-03-09T09:54:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350010140 2021-03-09T09:54:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bda0 con 0x7ff35000a6f0 2021-03-09T09:54:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071f0 con 0x7ff350015580 2021-03-09T09:54:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.156+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.160+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010140 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:54:27.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.364+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010140 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:54:27.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:27.768+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010140 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:54:28.573 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:28.572+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010140 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:54:28.769 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:28.761+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:30.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:30.176+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010140 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:54:31.726 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:31.724+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:31.727 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:31.724+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:31.728 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.724+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015580 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:31.728 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.724+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015580 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:31.728 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.724+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015580 0x7ff35000fa30 :-1 s=OPENED pgs=652 cs=1 l=1).handle_message read tag failed 2021-03-09T09:54:31.728 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.724+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015580 0x7ff35000fa30 :-1 s=OPENED pgs=652 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:54:31.729 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.724+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:54:31.775 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:31.773+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:31.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.772+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:31.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.772+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:31.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.772+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000a6f0 0x7ff35001cc00 :-1 s=OPENED pgs=650 cs=1 l=1).handle_message read tag failed 2021-03-09T09:54:31.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.772+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000a6f0 0x7ff35001cc00 :-1 s=OPENED pgs=650 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:54:31.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.772+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:54:31.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:31.772+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:54:33.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:33.380+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010140 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:54:33.789 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:33.785+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:36.727 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:36.724+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:36.727 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:36.724+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:36.775 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:36.772+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:38.792 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:38.784+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:39.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:39.784+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010140 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:54:41.727 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:41.723+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:41.727 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:41.723+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:41.775 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:41.768+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:43.813 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:43.808+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:46.727 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:46.723+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:46.728 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:46.723+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:46.776 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:46.768+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:48.816 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:48.812+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:51.727 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:51.723+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:51.728 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:51.723+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:51.776 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:51.768+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:52.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:52.599+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010140 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:54:53.837 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:53.832+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:56.727 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:54:56.723+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:54:56.728 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:54:56.723+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:54:56.776 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:54:56.768+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:54:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.159+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:54:57.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:54:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:54:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015580 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000a6f0 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010140 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,0,6,7,2,1,4,8,3] 2021-03-09T09:54:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000a6f0 addr v1:172.21.15.173:6791/0 2021-03-09T09:54:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350015580 addr v1:172.21.15.143:6789/0 2021-03-09T09:54:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T09:54:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015990 con 0x7ff350015580 2021-03-09T09:54:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff35000a6f0 2021-03-09T09:54:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071f0 con 0x7ff3500163b0 2021-03-09T09:54:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.163+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015580 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:54:57.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.367+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015580 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:54:57.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:57.771+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015580 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:54:58.576 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:58.575+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015580 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:54:58.840 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:54:58.836+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:54:58.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:58.835+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:58.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:58.835+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:58.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:58.835+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 0x7ff350006a90 :-1 s=OPENED pgs=662 cs=1 l=1).handle_message read tag failed 2021-03-09T09:54:58.842 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:58.839+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 0x7ff350006a90 :-1 s=OPENED pgs=662 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:54:58.842 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:54:58.839+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:55:00.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:00.179+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015580 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:55:01.728 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:01.727+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:01.728 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:01.727+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:01.729 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:01.727+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:55:01.730 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:01.727+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:01.730 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:01.727+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=677 cs=1 l=1).handle_message read tag failed 2021-03-09T09:55:01.730 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:01.727+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=677 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:55:01.731 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:01.727+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:55:01.776 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:01.772+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:03.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:03.383+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015580 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:55:03.862 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:03.856+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:06.728 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:06.723+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:06.728 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:06.723+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:06.776 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:06.772+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:08.863 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:08.860+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:09.791 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:09.787+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015580 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:55:11.728 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:11.727+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:11.729 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:11.727+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:11.776 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:11.772+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:13.885 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:13.880+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:16.728 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:16.726+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:16.729 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:16.726+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:16.777 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:16.772+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:18.888 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:18.884+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:21.728 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:21.726+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:21.729 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:21.726+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:21.777 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:21.772+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:22.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:22.598+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015580 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:55:23.909 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:23.904+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:26.728 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:26.726+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:26.729 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:26.726+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:26.777 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:26.776+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:55:27.165 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:55:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:55:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015580 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,4,8,1,3,2] 2021-03-09T09:55:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T09:55:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001b820 addr v1:172.21.15.143:6791/0 2021-03-09T09:55:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350015170 addr v1:172.21.15.143:6793/0 2021-03-09T09:55:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bd60 con 0x7ff35001b820 2021-03-09T09:55:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015990 con 0x7ff3500111f0 2021-03-09T09:55:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071f0 con 0x7ff350015170 2021-03-09T09:55:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.162+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001b820 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:55:27.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.366+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001b820 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:55:27.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:27.770+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001b820 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:55:28.576 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:28.574+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001b820 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:55:28.912 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:28.908+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:30.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:30.178+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001b820 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:55:31.729 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:31.726+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:31.729 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:31.726+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:31.730 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.726+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:55:31.730 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.726+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:31.730 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.726+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 0x7ff350020ad0 :-1 s=OPENED pgs=677 cs=1 l=1).handle_message read tag failed 2021-03-09T09:55:31.731 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.726+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 0x7ff350020ad0 :-1 s=OPENED pgs=677 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:55:31.731 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.726+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:55:31.731 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.726+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:55:31.777 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:31.776+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:31.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.774+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:31.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.774+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:31.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.774+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35000fa30 :-1 s=OPENED pgs=680 cs=1 l=1).handle_message read tag failed 2021-03-09T09:55:31.779 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.774+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35000fa30 :-1 s=OPENED pgs=680 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:55:31.779 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:31.774+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:55:33.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:33.382+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001b820 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:55:33.933 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:33.932+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:36.729 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:36.726+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:36.730 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:36.726+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:36.777 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:36.776+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:38.935 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:38.931+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:39.791 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:39.786+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001b820 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:55:41.729 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:41.726+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:41.730 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:41.726+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:41.777 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:41.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:43.957 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:43.955+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:46.729 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:46.726+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:46.730 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:46.726+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:46.777 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:46.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:48.959 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:48.955+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:51.729 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:51.726+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:51.730 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:51.726+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:51.778 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:51.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:52.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:52.593+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001b820 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:55:53.981 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:53.979+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:55:56.730 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:55:56.725+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:55:56.730 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:55:56.725+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:55:56.778 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:55:56.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:55:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:55:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:55:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:55:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001b820 legacy=0x7ff35001cc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,2,1] 2021-03-09T09:55:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350015170 addr v1:172.21.15.173:6790/0 2021-03-09T09:55:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T09:55:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350015580 addr v1:172.21.15.173:6789/0 2021-03-09T09:55:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350015580 2021-03-09T09:55:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bd60 con 0x7ff3500163b0 2021-03-09T09:55:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015990 con 0x7ff350015170 2021-03-09T09:55:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:55:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:55:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.161+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:55:57.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.365+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:55:57.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.365+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:55:57.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.365+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:55:57.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.769+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:55:57.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.769+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:55:57.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:57.769+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:55:58.577 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:58.573+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:55:58.577 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:58.573+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:55:58.577 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:55:58.573+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:55:58.983 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:55:58.979+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:00.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:00.177+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:00.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:00.177+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:56:00.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:00.177+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:01.730 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:01.725+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:01.731 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:01.725+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:01.778 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:01.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:03.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:03.381+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:03.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:03.381+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:03.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:03.381+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:56:03.986 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:03.983+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:06.730 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:06.729+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:06.731 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:06.729+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:06.778 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:06.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:09.005 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:09.003+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:09.791 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:09.789+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:09.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:09.789+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:56:09.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:09.789+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:11.730 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:11.729+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:11.731 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:11.729+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:11.778 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:11.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:14.007 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:14.003+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:16.730 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:16.725+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:16.731 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:16.725+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:16.778 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:16.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:19.029 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:19.027+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:21.731 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:21.725+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:21.731 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:21.725+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:21.779 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:21.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:22.601 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:22.597+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:22.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:22.601+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:56:22.605 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:22.601+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:24.032 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:24.027+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:26.731 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:26.729+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:26.731 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:26.729+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:26.779 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:26.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:56:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:56:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:56:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35001c4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,2,5,6,3,4,8,7,0] 2021-03-09T09:56:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:56:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350015580 addr v1:172.21.15.173:6789/0 2021-03-09T09:56:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T09:56:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500111f0 addr v1:172.21.15.173:6791/0 2021-03-09T09:56:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071c0 con 0x7ff350015580 2021-03-09T09:56:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500163b0 2021-03-09T09:56:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bd60 con 0x7ff3500111f0 2021-03-09T09:56:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.161+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:27.366 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.365+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:27.367 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.365+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:27.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.769+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:27.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:27.769+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:28.574 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:28.573+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:28.574 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:28.573+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:29.053 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:29.051+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:29.054 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:29.049+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:56:29.054 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:29.049+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:29.055 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:29.049+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 0x7ff35001b110 :-1 s=OPENED pgs=708 cs=1 l=1).handle_message read tag failed 2021-03-09T09:56:29.055 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:29.049+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 0x7ff35001b110 :-1 s=OPENED pgs=708 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:56:29.055 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:29.049+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:56:30.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:30.176+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:30.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:30.176+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:31.731 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:31.728+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:31.732 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:31.728+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:31.779 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:31.775+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:33.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:33.380+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:33.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:33.380+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:34.055 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:34.051+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:36.731 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:36.728+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:36.732 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:36.728+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:36.779 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:36.779+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:39.077 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:39.074+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:39.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:39.784+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:39.790 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:39.784+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:41.732 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:41.728+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:41.735 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:41.728+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:41.779 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:41.778+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:44.079 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:44.074+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:46.731 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:46.728+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:46.732 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:46.728+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:46.779 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:46.774+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:49.101 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:49.098+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:51.732 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:51.728+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:51.732 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:51.728+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:51.780 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:51.778+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:52.594 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:52.592+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:56:52.602 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:52.600+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:56:54.103 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:54.098+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:56:56.732 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:56:56.728+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:56:56.733 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:56:56.728+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:56:56.780 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:56:56.774+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:56:57.166 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:56:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:56:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:56:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015580 legacy=0x7ff35000be10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,8,0,4,7,6] 2021-03-09T09:56:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35001b820 addr v1:172.21.15.173:6790/0 2021-03-09T09:56:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35000a6f0 addr v1:172.21.15.143:6793/0 2021-03-09T09:56:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350012150 addr v1:172.21.15.143:6789/0 2021-03-09T09:56:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015990 con 0x7ff350012150 2021-03-09T09:56:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071c0 con 0x7ff35001b820 2021-03-09T09:56:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff35000a6f0 2021-03-09T09:56:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350012150 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:56:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.164+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001b820 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:56:57.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.368+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350012150 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:56:57.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.368+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001b820 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:56:57.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.772+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350012150 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:56:57.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:57.772+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001b820 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:56:58.577 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:58.576+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350012150 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:56:58.577 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:56:58.576+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001b820 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:56:59.125 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:56:59.122+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:00.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:00.180+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350012150 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:57:00.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:00.180+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001b820 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:57:01.732 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:01.728+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:01.733 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:01.728+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:01.733 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:01.728+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:01.734 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:01.728+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:01.734 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:01.728+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 0x7ff350010130 :-1 s=OPENED pgs=721 cs=1 l=1).handle_message read tag failed 2021-03-09T09:57:01.734 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:01.728+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 0x7ff350010130 :-1 s=OPENED pgs=721 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:57:01.734 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:01.728+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:57:01.780 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:01.774+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:03.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:03.384+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350012150 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:57:03.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:03.384+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001b820 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:57:04.127 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:04.122+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:06.732 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:06.727+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:06.733 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:06.727+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:06.780 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:06.778+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:09.149 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:09.146+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:09.791 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:09.787+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350012150 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:57:09.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:09.787+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001b820 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:57:11.733 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:11.731+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:11.733 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:11.731+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:11.780 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:11.778+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:14.151 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:14.146+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:16.733 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:16.731+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:16.733 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:16.731+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:16.780 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:16.778+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:19.173 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:19.170+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:21.733 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:21.731+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:21.733 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:21.731+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:21.780 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:21.778+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:22.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:22.599+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350012150 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:57:22.605 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:22.603+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001b820 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:57:24.175 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:24.170+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:26.733 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:26.731+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:26.734 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:26.731+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:26.781 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:26.778+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:57:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:57:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:57:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001b820 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350012150 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:57:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,6,4] 2021-03-09T09:57:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T09:57:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350012150 addr v1:172.21.15.143:6792/0 2021-03-09T09:57:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500163b0 addr v1:172.21.15.143:6791/0 2021-03-09T09:57:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500163b0 2021-03-09T09:57:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015990 con 0x7ff350012150 2021-03-09T09:57:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071c0 con 0x7ff3500111f0 2021-03-09T09:57:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.163+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:27.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.367+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:27.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:27.771+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:28.576 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:28.575+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:29.197 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:29.194+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:30.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:30.179+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:31.733 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:31.731+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:31.734 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:31.731+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:31.735 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.731+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350012150 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:31.735 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.731+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350012150 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:31.735 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.731+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350012150 0x7ff35000ce10 :-1 s=OPENED pgs=747 cs=1 l=1).handle_message read tag failed 2021-03-09T09:57:31.736 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.731+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350012150 0x7ff35000ce10 :-1 s=OPENED pgs=747 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:57:31.736 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.731+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:57:31.781 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:31.778+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:31.782 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.779+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:31.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.779+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:31.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.779+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a90 :-1 s=OPENED pgs=735 cs=1 l=1).handle_message read tag failed 2021-03-09T09:57:31.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.779+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a90 :-1 s=OPENED pgs=735 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:57:31.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:31.779+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:57:33.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:33.383+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:34.199 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:34.194+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:36.733 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:36.731+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:36.734 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:36.731+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:36.781 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:36.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:39.221 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:39.217+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:39.791 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:39.787+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:41.734 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:41.731+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:41.734 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:41.731+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:41.781 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:41.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:44.223 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:44.217+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:46.734 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:46.730+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:46.735 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:46.730+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:46.781 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:46.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:49.245 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:49.241+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:51.734 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:51.730+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:51.735 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:51.730+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:51.782 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:51.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:52.602 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:52.598+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:54.247 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:54.241+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:56.734 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:57:56.730+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:57:56.735 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:57:56.730+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:57:56.782 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:57:56.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:57:57.167 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:57:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:57:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:57:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350012150 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,7,4,3,6,8,2,1,0] 2021-03-09T09:57:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000a6f0 addr v1:172.21.15.173:6791/0 2021-03-09T09:57:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T09:57:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500163b0 addr v1:172.21.15.143:6791/0 2021-03-09T09:57:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d6b0 con 0x7ff3500163b0 2021-03-09T09:57:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff35000a6f0 2021-03-09T09:57:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071c0 con 0x7ff3500111f0 2021-03-09T09:57:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.162+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:57.368 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.366+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:57.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:57.770+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:58.576 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:58.574+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:57:59.269 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:57:59.265+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:57:59.270 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:59.266+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:59.270 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:59.266+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:59.270 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:59.266+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 0x7ff350010130 :-1 s=OPENED pgs=752 cs=1 l=1).handle_message read tag failed 2021-03-09T09:57:59.271 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:59.266+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 0x7ff350010130 :-1 s=OPENED pgs=752 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:57:59.271 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:57:59.266+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:58:00.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:00.178+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:58:01.735 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:01.730+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:01.735 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:01.730+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:01.782 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:01.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:01.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:01.778+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:58:01.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:01.778+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:01.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:01.778+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a90 :-1 s=OPENED pgs=752 cs=1 l=1).handle_message read tag failed 2021-03-09T09:58:01.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:01.778+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a90 :-1 s=OPENED pgs=752 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:58:01.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:01.778+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T09:58:03.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:03.382+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:58:04.271 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:04.265+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:06.735 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:06.734+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:06.735 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:06.734+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:06.782 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:06.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:09.293 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:09.289+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:09.791 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:09.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:58:11.735 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:11.734+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:11.735 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:11.734+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:11.782 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:11.781+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:14.295 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:14.289+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:16.735 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:16.734+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:16.736 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:16.734+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:16.782 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:16.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:19.317 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:19.313+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:21.735 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:21.733+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:21.736 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:21.733+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:21.782 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:21.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:22.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:22.601+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T09:58:24.319 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:24.313+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:26.736 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:26.733+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:26.736 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:26.733+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:26.783 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:26.777+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:58:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:58:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:58:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,8,1,3,6,2] 2021-03-09T09:58:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350015550 addr v1:172.21.15.143:6789/0 2021-03-09T09:58:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35000fa30 addr v1:172.21.15.143:6793/0 2021-03-09T09:58:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001eac0 addr v1:172.21.15.173:6789/0 2021-03-09T09:58:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350015550 2021-03-09T09:58:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d6b0 con 0x7ff35001eac0 2021-03-09T09:58:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff35000fa30 2021-03-09T09:58:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015550 legacy=0x7ff350003a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:58:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.165+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:58:27.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.369+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015550 legacy=0x7ff350003a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:58:27.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.369+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:58:27.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.773+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015550 legacy=0x7ff350003a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:58:27.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:27.773+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:58:28.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:28.577+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015550 legacy=0x7ff350003a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:58:28.582 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:28.577+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:58:29.341 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:29.337+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:30.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:30.181+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015550 legacy=0x7ff350003a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:58:30.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:30.181+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:58:31.736 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:31.733+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:31.737 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:31.733+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:31.738 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:31.733+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000fa30 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:31.738 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:31.733+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000fa30 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:31.738 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:31.733+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000fa30 0x7ff35001b110 :-1 s=OPENED pgs=776 cs=1 l=1).handle_message read tag failed 2021-03-09T09:58:31.738 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:31.733+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000fa30 0x7ff35001b110 :-1 s=OPENED pgs=776 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:58:31.739 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:31.733+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:58:31.739 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:31.733+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T09:58:31.783 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:31.781+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:33.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:33.385+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015550 legacy=0x7ff350003a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:58:33.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:33.385+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:58:34.343 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:34.337+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:36.736 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:36.729+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:36.737 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:36.729+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:36.783 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:36.776+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:39.365 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:39.360+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:39.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:39.789+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015550 legacy=0x7ff350003a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:58:39.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:39.789+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:58:41.736 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:41.733+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:41.737 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:41.733+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:41.783 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:41.780+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:44.367 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:44.364+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:46.737 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:46.733+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:46.737 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:46.733+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:46.783 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:46.776+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:49.389 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:49.384+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:51.737 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:51.733+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:51.737 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:51.733+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:51.784 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:51.780+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:52.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:52.597+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015550 legacy=0x7ff350003a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T09:58:52.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:52.605+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:58:54.391 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:54.384+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:58:56.737 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:58:56.733+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:58:56.737 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:58:56.733+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:58:56.784 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:58:56.780+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:58:57.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:58:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:58:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:58:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000fa30 legacy=0x7ff35001b110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015550 legacy=0x7ff350003a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,6,2] 2021-03-09T09:58:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35000fa30 addr v1:172.21.15.173:6790/0 2021-03-09T09:58:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T09:58:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T09:58:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015930 con 0x7ff3500111f0 2021-03-09T09:58:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff35000fa30 2021-03-09T09:58:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d6b0 con 0x7ff3500163b0 2021-03-09T09:58:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:58:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.165+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000fa30 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:58:57.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.369+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:58:57.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.369+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000fa30 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:58:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.773+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:58:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:57.773+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000fa30 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:58:58.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:58.577+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:58:58.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:58:58.577+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000fa30 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:58:59.393 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:58:59.388+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:00.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:00.180+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:59:00.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:00.180+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000fa30 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:01.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:01.732+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:01.737 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:01.732+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:01.738 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:01.732+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:01.739 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:01.732+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=802 cs=1 l=1).handle_message read tag failed 2021-03-09T09:59:01.739 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:01.732+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=802 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:59:01.739 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:01.732+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:01.740 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:01.732+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:59:01.784 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:01.780+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:03.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:03.384+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:59:03.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:03.384+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000fa30 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:04.412 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:04.408+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:06.737 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:06.732+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:06.738 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:06.732+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:06.784 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:06.784+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:09.415 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:09.408+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:09.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:09.792+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:59:09.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:09.792+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000fa30 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:11.737 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:11.736+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:11.738 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:11.736+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:11.784 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:11.784+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:14.436 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:14.432+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:16.738 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:16.736+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:16.738 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:16.736+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:16.784 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:16.780+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:19.439 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:19.432+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:21.738 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:21.736+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:21.738 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:21.736+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:21.785 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:21.784+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:22.605 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:22.604+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:59:22.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:22.604+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000fa30 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:24.460 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:24.460+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:26.738 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:26.736+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:26.739 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:26.736+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:26.785 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:26.784+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:59:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:59:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:59:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000fa30 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T09:59:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,6,5,8,0,7,2,1,4] 2021-03-09T09:59:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T09:59:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T09:59:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000a6f0 addr v1:172.21.15.173:6791/0 2021-03-09T09:59:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500111f0 2021-03-09T09:59:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015930 con 0x7ff35000a6f0 2021-03-09T09:59:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500163b0 2021-03-09T09:59:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.164+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:27.369 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.368+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:27.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:27.772+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:28.577 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:28.576+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:29.463 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:29.460+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:29.464 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:29.460+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:59:29.464 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:29.460+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:29.464 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:29.460+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 0x7ff35000fa20 :-1 s=OPENED pgs=802 cs=1 l=1).handle_message read tag failed 2021-03-09T09:59:29.465 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:29.460+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 0x7ff35000fa20 :-1 s=OPENED pgs=802 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:59:29.465 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:29.460+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T09:59:30.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:30.180+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:31.738 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:31.736+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:31.739 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:31.736+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:31.740 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:31.736+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:31.740 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:31.736+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:31.740 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:31.736+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=820 cs=1 l=1).handle_message read tag failed 2021-03-09T09:59:31.740 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:31.736+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=820 cs=1 l=1).fault on lossy channel, failing 2021-03-09T09:59:31.741 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:31.736+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T09:59:31.785 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:31.784+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:33.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:33.384+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:34.484 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:34.483+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:36.738 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:36.736+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:36.739 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:36.736+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:36.785 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:36.783+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:39.486 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:39.483+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:39.791 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:39.787+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:41.738 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:41.735+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:41.739 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:41.735+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:41.786 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:41.783+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:44.508 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:44.507+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:46.739 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:46.735+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:46.739 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:46.735+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:46.786 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:46.783+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:49.511 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:49.507+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:51.739 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:51.735+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:51.740 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:51.735+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:51.786 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:51.783+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:52.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:52.603+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T09:59:54.532 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:54.531+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T09:59:56.739 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T09:59:56.735+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T09:59:56.740 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T09:59:56.735+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T09:59:56.786 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T09:59:56.783+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T09:59:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T09:59:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T09:59:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:59:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,7,2,0,4,8] 2021-03-09T09:59:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350010130 addr v1:172.21.15.173:6789/0 2021-03-09T09:59:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350013870 addr v1:172.21.15.173:6792/0 2021-03-09T09:59:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500153a0 addr v1:172.21.15.143:6790/0 2021-03-09T09:59:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d6b0 con 0x7ff350010130 2021-03-09T09:59:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500153a0 2021-03-09T09:59:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bd40 con 0x7ff350013870 2021-03-09T09:59:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500153a0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:59:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.167+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:59:57.372 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.371+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500153a0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:59:57.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.371+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:59:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.775+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500153a0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:59:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:57.775+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:59:58.580 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:58.579+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500153a0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T09:59:58.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T09:59:58.579+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T09:59:59.535 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T09:59:59.531+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:00.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:00.183+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500153a0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:00:00.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:00.183+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:00:01.739 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:01.735+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:01.740 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:01.735+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:01.786 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:01.783+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:01.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:01.783+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350013870 legacy=0x7ff350013c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:01.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:01.783+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350013870 legacy=0x7ff350013c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:01.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:01.783+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350013870 0x7ff350013c80 :-1 s=OPENED pgs=830 cs=1 l=1).handle_message read tag failed 2021-03-09T10:00:01.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:01.783+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350013870 0x7ff350013c80 :-1 s=OPENED pgs=830 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:00:01.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:01.783+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:00:03.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:03.387+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500153a0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:00:03.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:03.387+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:00:04.556 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:04.555+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:06.740 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:06.739+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:06.740 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:06.739+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:06.786 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:06.783+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:09.559 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:09.555+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:09.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:09.791+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500153a0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:00:09.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:09.791+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:00:11.740 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:11.739+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:11.741 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:11.739+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:11.787 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:11.787+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:14.580 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:14.579+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:16.740 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:16.735+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:16.741 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:16.735+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:16.787 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:16.783+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:19.583 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:19.579+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:21.740 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:21.738+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:21.741 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:21.738+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:21.787 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:21.787+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:22.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:22.606+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500153a0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:00:22.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:22.606+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:00:24.604 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:24.603+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:26.740 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:26.738+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:26.741 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:26.738+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:26.787 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:26.783+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.162+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:00:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.162+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:00:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.162+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:00:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350013870 legacy=0x7ff350013c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.162+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500153a0 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:00:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,8,4] 2021-03-09T10:00:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:00:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T10:00:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500163b0 addr v1:172.21.15.143:6793/0 2021-03-09T10:00:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35000a6f0 addr v1:172.21.15.143:6791/0 2021-03-09T10:00:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500157f0 con 0x7ff3500111f0 2021-03-09T10:00:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d6b0 con 0x7ff35000a6f0 2021-03-09T10:00:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:00:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500163b0 2021-03-09T10:00:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.166+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:00:27.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.370+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:00:27.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.370+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:00:27.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.774+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:00:27.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:27.774+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:00:28.580 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:28.578+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:00:28.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:28.578+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:00:29.607 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:29.602+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:30.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:30.182+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:00:30.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:30.182+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:00:31.741 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:31.738+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:31.741 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:31.738+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:31.742 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:31.738+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:31.742 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:31.738+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:31.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:31.738+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=852 cs=1 l=1).handle_message read tag failed 2021-03-09T10:00:31.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:31.738+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=852 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:00:31.743 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:31.738+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:00:31.787 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:31.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:33.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:33.386+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:00:33.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:33.386+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:00:34.628 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:34.626+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:36.741 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:36.738+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:36.741 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:36.738+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:36.787 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:36.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:39.631 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:39.626+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:39.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:39.794+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:00:39.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:39.794+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:00:41.741 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:41.738+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:41.742 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:41.738+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:41.788 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:41.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:44.652 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:44.650+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:46.741 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:46.738+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:46.742 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:46.738+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:46.788 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:46.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:49.655 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:49.650+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:51.741 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:51.738+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:51.742 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:51.738+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:51.788 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:51.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:52.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:52.602+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:00:52.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:52.602+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:00:54.676 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:54.674+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:00:56.742 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:00:56.737+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:00:56.742 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:00:56.737+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:00:56.788 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:00:56.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:00:57.169 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:00:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:00:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:00:57.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,1,2,4,5,0,8,3,6] 2021-03-09T10:00:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350010130 addr v1:172.21.15.173:6792/0 2021-03-09T10:00:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350013870 addr v1:172.21.15.173:6789/0 2021-03-09T10:00:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff350015370 addr v1:172.21.15.143:6790/0 2021-03-09T10:00:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bca0 con 0x7ff350013870 2021-03-09T10:00:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500157f0 con 0x7ff350015370 2021-03-09T10:00:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d6b0 con 0x7ff350010130 2021-03-09T10:00:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015370 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:00:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.165+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013870 legacy=0x7ff350013c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:00:57.372 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.369+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015370 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:00:57.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.369+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013870 legacy=0x7ff350013c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:00:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.773+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015370 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:00:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:57.773+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013870 legacy=0x7ff350013c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:00:58.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:58.577+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015370 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:00:58.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:00:58.577+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013870 legacy=0x7ff350013c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:00:59.679 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:00:59.674+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:00.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:00.181+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015370 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:01:00.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:00.181+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013870 legacy=0x7ff350013c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:01:01.742 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:01.737+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:01.743 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:01.737+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:01.789 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:01.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:01.790 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:01.785+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:01.790 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:01.785+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:01.790 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:01.785+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350010130 0x7ff350012150 :-1 s=OPENED pgs=874 cs=1 l=1).handle_message read tag failed 2021-03-09T10:01:01.790 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:01.785+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350010130 0x7ff350012150 :-1 s=OPENED pgs=874 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:01:01.790 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:01.785+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:01:03.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:03.385+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015370 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:01:03.390 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:03.385+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013870 legacy=0x7ff350013c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:01:04.704 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:04.702+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:06.742 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:06.737+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:06.743 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:06.737+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:06.789 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:06.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:09.706 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:09.702+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:09.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:09.793+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015370 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:01:09.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:09.793+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013870 legacy=0x7ff350013c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:01:11.742 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:11.737+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:11.743 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:11.737+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:11.789 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:11.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:14.728 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:14.726+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:16.742 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:16.737+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:16.743 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:16.737+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:16.789 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:16.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:19.730 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:19.726+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:21.743 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:21.741+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:21.743 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:21.741+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:21.789 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:21.786+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:22.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:22.605+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015370 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:01:22.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:22.605+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013870 legacy=0x7ff350013c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:01:24.752 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:24.750+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:26.743 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:26.741+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:26.743 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:26.741+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:26.789 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:26.785+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:01:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:01:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:01:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015370 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350013870 legacy=0x7ff350013c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:01:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,0,6,8,5,4] 2021-03-09T10:01:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:01:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:01:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350015370 addr v1:172.21.15.143:6789/0 2021-03-09T10:01:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T10:01:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350015370 2021-03-09T10:01:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bca0 con 0x7ff3500111f0 2021-03-09T10:01:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:01:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500157f0 con 0x7ff3500163b0 2021-03-09T10:01:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.165+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015370 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:01:27.370 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.369+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:01:27.370 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.369+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015370 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:01:27.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.773+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015370 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:01:27.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:27.773+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:01:28.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:28.577+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015370 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:01:28.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:28.577+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:01:29.754 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:29.749+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:30.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:30.181+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015370 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:01:30.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:30.181+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:01:31.743 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:31.737+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:31.744 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:31.737+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:31.744 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:31.737+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:01:31.745 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:31.737+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:31.745 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:31.737+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000fa20 :-1 s=OPENED pgs=910 cs=1 l=1).handle_message read tag failed 2021-03-09T10:01:31.745 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:31.737+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000fa20 :-1 s=OPENED pgs=910 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:01:31.745 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:31.737+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:01:31.790 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:31.785+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:33.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:33.385+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015370 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:01:33.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:33.385+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:01:34.776 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:34.773+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:36.743 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:36.740+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:36.744 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:36.740+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:36.790 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:36.789+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:39.778 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:39.773+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:39.794 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:39.788+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015370 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:01:39.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:39.788+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:01:41.743 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:41.740+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:41.744 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:41.740+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:41.790 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:41.785+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:44.800 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:44.797+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:46.744 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:46.736+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:46.744 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:46.736+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:46.790 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:46.785+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:49.802 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:49.797+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:51.744 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:51.740+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:51.744 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:51.740+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:51.790 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:51.785+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:52.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:52.600+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015370 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:01:52.605 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:52.600+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:01:54.824 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:54.821+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:56.744 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:01:56.740+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:01:56.744 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:01:56.740+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:01:56.790 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:01:56.785+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:01:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:01:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:01:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:01:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015370 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,8,5] 2021-03-09T10:01:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35000a6f0 addr v1:172.21.15.143:6791/0 2021-03-09T10:01:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350013160 addr v1:172.21.15.143:6793/0 2021-03-09T10:01:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350014c90 addr v1:172.21.15.173:6791/0 2021-03-09T10:01:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff35000a6f0 2021-03-09T10:01:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350014c90 2021-03-09T10:01:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bca0 con 0x7ff350013160 2021-03-09T10:01:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.168+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:01:57.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.372+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:01:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:57.776+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:01:58.585 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:58.580+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:01:59.844 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:01:59.841+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:01:59.844 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:59.836+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014c90 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:01:59.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:59.836+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014c90 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:59.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:59.836+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014c90 0x7ff35001b110 :-1 s=OPENED pgs=922 cs=1 l=1).handle_message read tag failed 2021-03-09T10:01:59.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:59.836+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014c90 0x7ff35001b110 :-1 s=OPENED pgs=922 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:01:59.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:01:59.836+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:02:00.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:00.184+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:01.744 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:01.740+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:01.745 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:01.744+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:01.746 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:01.744+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350013160 legacy=0x7ff350013570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:01.746 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:01.744+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350013160 legacy=0x7ff350013570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:01.746 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:01.744+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350013160 0x7ff350013570 :-1 s=OPENED pgs=909 cs=1 l=1).handle_message read tag failed 2021-03-09T10:02:01.746 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:01.744+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350013160 0x7ff350013570 :-1 s=OPENED pgs=909 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:02:01.747 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:01.744+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:02:01.747 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:01.744+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:02:01.791 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:01.785+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:03.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:03.388+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:04.846 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:04.841+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:06.744 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:06.744+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:06.745 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:06.744+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:06.791 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:06.785+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:09.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:09.792+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:09.868 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:09.865+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:11.744 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:11.744+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:11.745 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:11.744+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:11.791 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:11.789+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:14.870 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:14.865+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:16.745 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:16.743+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:16.745 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:16.743+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:16.791 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:16.789+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:19.892 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:19.889+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:21.745 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:21.743+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:21.746 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:21.743+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:21.791 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:21.788+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:22.606 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:22.603+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:24.894 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:24.892+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:26.745 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:26.743+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:26.746 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:26.743+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:26.795 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:26.788+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:02:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:02:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:02:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350013160 legacy=0x7ff350013570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014c90 legacy=0x7ff35001b110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,4,3,5,6,8,2,0,7] 2021-03-09T10:02:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350014c90 addr v1:172.21.15.173:6789/0 2021-03-09T10:02:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35000a6f0 addr v1:172.21.15.143:6791/0 2021-03-09T10:02:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350013160 addr v1:172.21.15.173:6790/0 2021-03-09T10:02:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500157f0 con 0x7ff350014c90 2021-03-09T10:02:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff350013160 2021-03-09T10:02:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff35000a6f0 2021-03-09T10:02:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014c90 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:02:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.167+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff350015070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:02:27.372 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.371+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:27.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.371+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff350015070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:02:27.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.371+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014c90 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:02:27.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.775+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:27.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.775+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014c90 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:02:27.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:27.775+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff350015070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:02:28.580 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:28.579+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:28.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:28.579+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff350015070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:02:28.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:28.579+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014c90 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:02:29.916 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:29.912+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:30.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:30.183+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:30.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:30.183+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff350015070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:02:30.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:30.183+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014c90 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:02:31.745 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:31.743+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:31.746 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:31.743+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:31.792 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:31.788+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:33.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:33.387+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:33.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:33.387+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014c90 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:02:33.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:33.387+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff350015070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:02:34.918 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:34.916+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:36.745 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:36.743+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:36.746 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:36.743+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:36.792 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:36.792+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:39.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:39.791+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:39.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:39.791+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014c90 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:02:39.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:39.791+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff350015070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:02:39.940 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:39.936+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:41.746 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:41.743+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:41.747 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:41.743+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:41.792 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:41.792+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:44.942 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:44.940+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:46.746 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:46.743+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:46.746 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:46.743+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:46.792 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:46.788+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:49.964 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:49.960+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:51.746 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:51.743+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:51.747 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:51.743+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:51.792 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:51.792+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:52.605 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:52.603+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:02:52.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:52.607+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff350015070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:02:52.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:52.607+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014c90 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:02:54.966 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:54.964+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:56.746 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:02:56.742+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:02:56.747 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:02:56.742+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:02:56.792 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:02:56.792+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:02:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:02:57.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:02:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:02:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff350015070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014c90 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,5,2,6,7,8] 2021-03-09T10:02:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T10:02:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500163b0 addr v1:172.21.15.173:6791/0 2021-03-09T10:02:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff350014550 addr v1:172.21.15.143:6790/0 2021-03-09T10:02:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bca0 con 0x7ff3500111f0 2021-03-09T10:02:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500157f0 con 0x7ff350014550 2021-03-09T10:02:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff3500163b0 2021-03-09T10:02:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:02:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.166+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350014550 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:02:57.372 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.370+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350014550 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:02:57.373 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.370+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:02:57.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.774+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:02:57.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:57.774+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350014550 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:02:58.580 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:58.578+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:02:58.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:58.578+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350014550 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:02:59.988 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:02:59.984+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:02:59.989 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:59.982+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:59.989 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:59.982+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:59.989 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:59.982+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350010130 :-1 s=OPENED pgs=955 cs=1 l=1).handle_message read tag failed 2021-03-09T10:02:59.989 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:59.982+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350010130 :-1 s=OPENED pgs=955 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:02:59.990 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:02:59.982+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:03:00.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:00.182+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350014550 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:03:00.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:00.182+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:03:01.746 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:01.742+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:01.747 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:01.742+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:01.793 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:01.792+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:03.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:03.386+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350014550 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:03:03.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:03.386+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:03:04.990 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:04.988+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:06.747 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:06.742+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:06.747 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:06.742+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:06.793 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:06.792+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:09.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:09.790+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:03:09.794 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:09.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350014550 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:03:10.015 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:10.016+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:11.747 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:11.746+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:11.747 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:11.746+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:11.793 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:11.792+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:15.018 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:15.016+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:16.747 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:16.742+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:16.748 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:16.742+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:16.793 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:16.792+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:20.040 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:20.039+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:21.747 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:21.746+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:21.748 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:21.746+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:21.793 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:21.791+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:22.605 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:22.602+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:03:22.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:22.602+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350014550 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:03:25.042 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:25.039+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:26.747 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:26.746+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:26.748 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:26.746+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:26.793 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:26.791+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:27.171 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:03:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:03:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:03:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350014550 legacy=0x7ff35001b110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,7,6] 2021-03-09T10:03:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:03:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001eac0 addr v1:172.21.15.143:6793/0 2021-03-09T10:03:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:03:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T10:03:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500163b0 2021-03-09T10:03:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bca0 con 0x7ff3500111f0 2021-03-09T10:03:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500157f0 con 0x7ff35001eac0 2021-03-09T10:03:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:27.170+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:30.063 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:30.063+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:31.748 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:31.742+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:31.748 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:31.742+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:31.749 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:31.750 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:31.750 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 0x7ff350020eb0 :-1 s=OPENED pgs=963 cs=1 l=1).handle_message read tag failed 2021-03-09T10:03:31.750 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 0x7ff350020eb0 :-1 s=OPENED pgs=963 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:03:31.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:03:31.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:31.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=985 cs=1 l=1).handle_message read tag failed 2021-03-09T10:03:31.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=985 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:03:31.752 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:03:31.752 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.742+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:03:31.794 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:31.791+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:31.794 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.794+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:31.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.794+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:31.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.794+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=972 cs=1 l=1).handle_message read tag failed 2021-03-09T10:03:31.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.794+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=972 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:03:31.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:31.794+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:03:35.066 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:35.063+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:36.748 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:36.741+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:36.748 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:36.741+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:36.794 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:36.791+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:40.087 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:40.087+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:41.748 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:41.745+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:41.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:41.745+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:41.794 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:41.791+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:45.090 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:45.087+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:46.748 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:46.745+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:46.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:46.745+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:46.794 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:46.791+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:50.111 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:50.111+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:51.748 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:51.745+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:51.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:51.745+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:51.794 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:51.795+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:55.114 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:03:55.111+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:03:56.749 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:03:56.745+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:03:56.750 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:03:56.745+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:03:56.795 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:03:56.795+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:03:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:03:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:03:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:03:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,0,8,7,1,5,3,4,2] 2021-03-09T10:03:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T10:03:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500096e0 addr v1:172.21.15.143:6789/0 2021-03-09T10:03:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001eac0 addr v1:172.21.15.143:6793/0 2021-03-09T10:03:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff3500096e0 2021-03-09T10:03:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500163b0 2021-03-09T10:03:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bca0 con 0x7ff35001eac0 2021-03-09T10:03:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.169+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500096e0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:03:57.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.373+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500096e0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:03:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:57.777+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500096e0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:03:58.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:03:58.581+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500096e0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:04:00.135 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:00.135+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:00.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:00.185+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500096e0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:04:01.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:01.745+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:01.750 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:01.745+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:01.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:01.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:04:01.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:01.751 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=1002 cs=1 l=1).handle_message read tag failed 2021-03-09T10:04:01.752 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=1002 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:04:01.752 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:01.752 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 0x7ff350010130 :-1 s=OPENED pgs=979 cs=1 l=1).handle_message read tag failed 2021-03-09T10:04:01.752 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 0x7ff350010130 :-1 s=OPENED pgs=979 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:04:01.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:04:01.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:01.745+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:04:01.795 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:01.795+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:03.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:03.389+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500096e0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:04:05.138 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:05.135+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:06.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:06.745+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:06.749 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:06.745+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:06.795 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:06.795+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:09.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:09.793+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500096e0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:04:10.159 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:10.159+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:11.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:11.749+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:11.750 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:11.749+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:11.795 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:11.795+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:15.162 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:15.158+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:16.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:16.748+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:16.750 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:16.748+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:16.795 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:16.794+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:20.187 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:20.186+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:21.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:21.748+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:21.750 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:21.748+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:21.795 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:21.794+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:22.605 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:22.600+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500096e0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:04:25.190 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:25.186+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:26.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:26.748+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:26.750 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:26.748+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:26.796 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:26.794+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:04:27.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:04:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:04:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001eac0 legacy=0x7ff350010130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500096e0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,3,2,7,5,1] 2021-03-09T10:04:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350012150 addr v1:172.21.15.143:6791/0 2021-03-09T10:04:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350014c50 addr v1:172.21.15.173:6790/0 2021-03-09T10:04:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001b120 addr v1:172.21.15.143:6790/0 2021-03-09T10:04:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff35001b120 2021-03-09T10:04:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff350014c50 2021-03-09T10:04:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bca0 con 0x7ff350012150 2021-03-09T10:04:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:04:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001b120 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:04:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.168+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014c50 legacy=0x7ff350015060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:04:27.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.372+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001b120 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:04:27.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.372+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:04:27.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.372+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014c50 legacy=0x7ff350015060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:04:27.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.776+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:04:27.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.776+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001b120 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:04:27.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:27.776+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014c50 legacy=0x7ff350015060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:04:28.581 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:28.580+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:04:28.582 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:28.580+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001b120 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:04:28.582 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:28.580+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014c50 legacy=0x7ff350015060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:04:30.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:30.184+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:04:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:30.184+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001b120 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:04:30.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:30.184+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014c50 legacy=0x7ff350015060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:04:30.211 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:30.210+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:31.749 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:31.748+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:31.750 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:31.748+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:31.796 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:31.794+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:33.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:33.388+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:04:33.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:33.388+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001b120 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:04:33.391 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:33.388+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014c50 legacy=0x7ff350015060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:04:35.214 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:35.210+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:36.750 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:36.748+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:36.751 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:36.748+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:36.796 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:36.794+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:39.794 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:39.792+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001b120 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:04:39.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:39.792+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:04:39.797 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:39.792+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014c50 legacy=0x7ff350015060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:04:40.235 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:40.234+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:41.750 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:41.748+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:41.751 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:41.748+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:41.796 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:41.794+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:45.238 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:45.234+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:46.750 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:46.748+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:46.751 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:46.748+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:46.797 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:46.794+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:50.259 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:50.258+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:51.751 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:51.748+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:51.751 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:51.748+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:51.797 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:51.794+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:52.607 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:52.604+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:04:52.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:52.604+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001b120 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:04:52.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:52.604+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014c50 legacy=0x7ff350015060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:04:55.261 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:04:55.257+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:04:56.751 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:04:56.747+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:04:56.751 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:04:56.747+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:04:56.797 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:04:56.793+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:04:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:04:57.172 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:04:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:04:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014c50 legacy=0x7ff350015060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001b120 legacy=0x7ff350020eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,7,1] 2021-03-09T10:04:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:04:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500225d0 addr v1:172.21.15.173:6791/0 2021-03-09T10:04:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:04:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:04:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:04:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500163b0 addr v1:172.21.15.173:6789/0 2021-03-09T10:04:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500163b0 2021-03-09T10:04:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500225d0 2021-03-09T10:04:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.167+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff3500111f0 2021-03-09T10:04:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.171+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.171+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:04:57.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.375+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:04:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:57.779+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:04:58.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:04:58.583+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:00.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:00.187+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:00.264 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:00.261+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:00.265 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:00.263+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500225d0 legacy=0x7ff3500249c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:00.265 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:00.263+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500225d0 legacy=0x7ff3500249c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:00.265 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:00.263+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500225d0 0x7ff3500249c0 :-1 s=OPENED pgs=1021 cs=1 l=1).handle_message read tag failed 2021-03-09T10:05:00.265 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:00.263+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500225d0 0x7ff3500249c0 :-1 s=OPENED pgs=1021 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:05:00.266 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:00.263+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:05:01.751 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:01.747+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:01.751 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:01.747+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:01.797 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:01.797+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:01.798 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:01.795+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:01.798 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:01.795+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:01.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:01.795+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1026 cs=1 l=1).handle_message read tag failed 2021-03-09T10:05:01.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:01.795+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1026 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:05:01.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:01.795+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:05:03.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:03.391+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:05.284 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:05.281+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:06.751 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:06.747+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:06.752 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:06.747+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:06.797 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:06.797+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:09.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:09.795+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:10.286 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:10.281+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:11.751 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:11.747+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:11.752 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:11.747+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:11.798 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:11.793+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:15.307 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:15.305+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:16.751 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:16.747+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:16.752 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:16.747+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:16.798 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:16.793+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:20.310 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:20.305+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:21.752 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:21.751+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:21.781 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:21.751+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:21.801 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:21.797+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:22.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:22.607+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:25.331 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:25.329+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:26.752 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:26.747+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:26.752 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:26.747+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:26.798 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:26.793+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:05:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:05:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:05:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500225d0 legacy=0x7ff3500249c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,1,4,2,0,8,7,3,5] 2021-03-09T10:05:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001b120 addr v1:172.21.15.143:6792/0 2021-03-09T10:05:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500163b0 addr v1:172.21.15.173:6789/0 2021-03-09T10:05:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35000a6f0 addr v1:172.21.15.143:6791/0 2021-03-09T10:05:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bca0 con 0x7ff3500163b0 2021-03-09T10:05:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff35000a6f0 2021-03-09T10:05:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff35001b120 2021-03-09T10:05:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:05:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.171+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:27.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.375+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:05:27.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.375+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:27.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.779+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:05:27.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:27.779+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:28.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:28.583+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:05:28.585 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:28.583+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:30.187+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:05:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:30.187+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:30.334 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:30.329+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:31.752 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:31.751+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:31.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:31.751+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:31.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:31.751+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:31.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:31.751+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 0x7ff35000fa20 :-1 s=OPENED pgs=1049 cs=1 l=1).handle_message read tag failed 2021-03-09T10:05:31.753 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:31.751+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 0x7ff35000fa20 :-1 s=OPENED pgs=1049 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:05:31.754 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:31.751+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:05:31.754 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:31.747+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:31.798 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:31.793+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:32.295+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354043c70 legacy=0x7ff3540402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000714) for more than 900000000 us, fault. 2021-03-09T10:05:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:32.295+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354043c70 0x7ff3540402e0 :-1 s=OPENED pgs=42 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:05:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:32.295+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354043c70 legacy=0x7ff3540402e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:32.295+0000 7ff372ffd700 10 monclient(hunting): get_auth_request con 0x7ff354030290 auth_method 0 2021-03-09T10:05:33.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:33.390+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:05:33.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:33.390+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:35.356 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:35.352+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:36.752 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:36.750+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:36.753 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:36.750+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:36.799 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:36.796+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:39.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:39.798+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:05:39.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:39.798+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:40.358 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:40.356+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:41.752 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:41.750+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:41.753 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:41.750+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:41.799 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:41.796+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:45.380 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:45.376+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:46.752 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:46.750+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:46.753 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:46.750+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:46.799 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:46.796+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:50.382 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:50.380+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:51.753 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:51.750+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:51.753 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:51.750+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:51.799 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:51.796+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:52.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:52.610+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:05:52.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:52.610+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:05:55.404 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:05:55.400+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:05:56.753 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:05:56.750+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:05:56.753 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:05:56.750+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:05:56.800 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:05:56.800+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:05:57.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:05:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:05:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:05:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000a6f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350010130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,7,5,2,0,8] 2021-03-09T10:05:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:05:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350013160 addr v1:172.21.15.173:6790/0 2021-03-09T10:05:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500163b0 addr v1:172.21.15.173:6792/0 2021-03-09T10:05:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500111f0 addr v1:172.21.15.173:6791/0 2021-03-09T10:05:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350013160 2021-03-09T10:05:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bca0 con 0x7ff3500111f0 2021-03-09T10:05:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500163b0 2021-03-09T10:05:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.170+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:05:57.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.374+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:05:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:57.778+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:05:58.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:05:58.582+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:06:00.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:00.186+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:06:00.407 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:00.404+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:00.407 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:00.402+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:06:00.407 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:00.402+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:00.407 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:00.402+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 0x7ff35000ce10 :-1 s=OPENED pgs=1049 cs=1 l=1).handle_message read tag failed 2021-03-09T10:06:00.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:00.402+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 0x7ff35000ce10 :-1 s=OPENED pgs=1049 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:06:00.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:00.402+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:06:01.753 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:01.750+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:01.754 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:01.750+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:01.800 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:01.796+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:01.798+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:01.798+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:01.798+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1057 cs=1 l=1).handle_message read tag failed 2021-03-09T10:06:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:01.798+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1057 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:06:01.802 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:01.798+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:06:03.394 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:03.390+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:06:05.428 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:05.424+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:06.753 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:06.750+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:06.754 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:06.750+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:06.800 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:06.800+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:09.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:09.798+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:06:10.430 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:10.428+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:11.753 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:11.749+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:11.754 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:11.749+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:11.800 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:11.800+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:15.452 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:15.451+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:16.754 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:16.749+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:16.754 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:16.749+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:16.800 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:16.799+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:20.454 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:20.451+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:21.754 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:21.749+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:21.754 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:21.749+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:21.800 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:21.799+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:22.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:22.601+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:06:25.476 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:25.475+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:26.754 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:26.749+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:26.754 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:26.749+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:26.801 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:26.799+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:27.173 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:06:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:06:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:06:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:27.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350013160 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,8,0] 2021-03-09T10:06:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500096e0 addr v1:172.21.15.143:6790/0 2021-03-09T10:06:27.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001af30 addr v1:172.21.15.143:6793/0 2021-03-09T10:06:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350010130 addr v1:172.21.15.143:6789/0 2021-03-09T10:06:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff350010130 2021-03-09T10:06:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500096e0 2021-03-09T10:06:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500096e0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff35001af30 2021-03-09T10:06:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.169+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35001dab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:06:27.374 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.373+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500096e0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:27.374 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.373+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35001dab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:06:27.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.777+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500096e0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:27.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:27.777+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35001dab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:06:28.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:28.581+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500096e0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:28.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:28.581+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35001dab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:06:30.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:30.185+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500096e0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:30.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:30.185+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35001dab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:06:30.478 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:30.475+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:31.754 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:31.753+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:31.755 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:31.753+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:31.756 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:31.753+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001af30 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:31.756 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:31.753+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001af30 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:31.756 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:31.753+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001af30 0x7ff35000fa20 :-1 s=OPENED pgs=1057 cs=1 l=1).handle_message read tag failed 2021-03-09T10:06:31.756 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:31.753+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001af30 0x7ff35000fa20 :-1 s=OPENED pgs=1057 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:06:31.756 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:31.753+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:06:31.801 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:31.799+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:33.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:33.389+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500096e0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:33.393 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:33.389+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35001dab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:06:35.504 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:35.503+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:36.754 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:36.753+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:36.755 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:36.753+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:36.801 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:36.799+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:39.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:39.797+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500096e0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:39.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:39.797+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35001dab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:06:40.506 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:40.503+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:41.755 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:41.753+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:41.755 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:41.753+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:41.801 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:41.799+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:45.528 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:45.527+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:46.755 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:46.752+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:46.755 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:46.752+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:46.801 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:46.799+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:50.530 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:50.527+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:51.755 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:51.752+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:51.755 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:51.752+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:51.802 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:51.799+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:52.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:52.604+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500096e0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:52.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:52.604+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35001dab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:06:55.552 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:06:55.550+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:06:56.755 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:06:56.752+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:06:56.756 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:06:56.752+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:06:56.802 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:06:56.798+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:06:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:06:57.174 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:06:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:06:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001af30 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500096e0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35001dab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,4,6,7,3,8,1,0,5] 2021-03-09T10:06:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:06:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001f1d0 addr v1:172.21.15.143:6790/0 2021-03-09T10:06:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500111f0 addr v1:172.21.15.143:6791/0 2021-03-09T10:06:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T10:06:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b340 con 0x7ff35001f1d0 2021-03-09T10:06:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff3500111f0 2021-03-09T10:06:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500163b0 2021-03-09T10:06:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:06:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.172+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001f1d0 legacy=0x7ff3500215c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:57.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.376+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:06:57.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.376+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001f1d0 legacy=0x7ff3500215c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:57.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.780+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:06:57.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:57.780+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001f1d0 legacy=0x7ff3500215c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:06:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:58.584+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:06:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:06:58.584+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001f1d0 legacy=0x7ff3500215c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:07:00.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:00.188+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:07:00.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:00.188+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001f1d0 legacy=0x7ff3500215c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:07:00.554 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:00.550+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:01.755 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:01.752+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:01.756 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:01.752+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:01.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:01.752+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:01.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:01.752+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:01.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:01.752+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=1101 cs=1 l=1).handle_message read tag failed 2021-03-09T10:07:01.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:01.752+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff35000ce10 :-1 s=OPENED pgs=1101 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:07:01.757 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:01.752+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:07:01.802 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:01.798+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:03.393 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:03.392+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:07:03.394 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:03.392+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001f1d0 legacy=0x7ff3500215c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:07:05.576 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:05.574+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:06.756 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:06.752+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:06.756 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:06.752+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:06.802 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:06.802+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:09.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:09.796+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:07:09.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:09.796+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001f1d0 legacy=0x7ff3500215c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:07:10.578 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:10.574+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:11.756 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:11.752+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:11.756 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:11.752+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:11.803 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:11.798+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:15.604 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:15.602+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:16.756 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:16.756+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:16.756 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:16.756+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:16.803 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:16.798+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:20.606 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:20.602+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:21.756 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:21.756+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:21.757 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:21.756+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:21.803 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:21.802+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:22.613 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:22.611+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001f1d0 legacy=0x7ff3500215c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:07:22.613 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:22.611+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:07:25.628 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:25.626+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:26.756 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:26.755+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:26.757 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:26.755+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:26.803 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:26.802+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:07:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:07:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:07:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001f1d0 legacy=0x7ff3500215c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,1,3,5,0,8] 2021-03-09T10:07:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001dab0 addr v1:172.21.15.173:6792/0 2021-03-09T10:07:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350010130 addr v1:172.21.15.173:6789/0 2021-03-09T10:07:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350005aa0 addr v1:172.21.15.173:6790/0 2021-03-09T10:07:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350010130 2021-03-09T10:07:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b340 con 0x7ff350005aa0 2021-03-09T10:07:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a60 con 0x7ff35001dab0 2021-03-09T10:07:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:07:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.171+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:07:27.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.375+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:07:27.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.375+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:07:27.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.779+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:07:27.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:27.779+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:07:28.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:28.583+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:07:28.585 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:28.583+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:07:30.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:30.187+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:07:30.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:30.187+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:07:30.630 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:30.626+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:31.756 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:31.755+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:31.757 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:31.755+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:31.803 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:31.802+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:31.805 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:31.803+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001dab0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:31.805 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:31.803+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001dab0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:31.806 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:31.803+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001dab0 0x7ff35000fa20 :-1 s=OPENED pgs=1102 cs=1 l=1).handle_message read tag failed 2021-03-09T10:07:31.806 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:31.803+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001dab0 0x7ff35000fa20 :-1 s=OPENED pgs=1102 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:07:31.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:31.803+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:07:33.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:33.391+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:07:33.393 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:33.391+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:07:35.652 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:35.650+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:36.757 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:36.755+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:36.757 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:36.755+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:36.804 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:36.802+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:39.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:39.795+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:07:39.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:39.795+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:07:40.654 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:40.649+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:41.757 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:41.755+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:41.757 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:41.755+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:41.804 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:41.801+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:45.676 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:45.673+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:46.757 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:46.755+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:46.757 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:46.755+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:46.804 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:46.801+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:50.678 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:50.673+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:51.757 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:51.755+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:51.758 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:51.755+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:51.804 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:51.801+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:52.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:52.611+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:07:52.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:52.611+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:07:55.700 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:07:55.697+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:07:56.757 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:07:56.755+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:07:56.758 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:07:56.755+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:07:56.804 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:07:56.801+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:07:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:07:57.175 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:07:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:07:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001dab0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500170f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,8,0] 2021-03-09T10:07:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:07:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350005ed0 addr v1:172.21.15.173:6791/0 2021-03-09T10:07:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:07:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350005aa0 addr v1:172.21.15.143:6793/0 2021-03-09T10:07:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T10:07:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500111f0 2021-03-09T10:07:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350005ed0 2021-03-09T10:07:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b340 con 0x7ff350005aa0 2021-03-09T10:07:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.171+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:07:57.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.375+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:07:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:57.778+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:07:58.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:07:58.582+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:08:00.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:00.186+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:08:00.720 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:00.717+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:00.721 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:00.718+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:00.721 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:00.718+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:00.721 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:00.718+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 0x7ff35001a820 :-1 s=OPENED pgs=1105 cs=1 l=1).handle_message read tag failed 2021-03-09T10:08:00.722 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:00.718+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 0x7ff35001a820 :-1 s=OPENED pgs=1105 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:08:00.722 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:00.718+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:08:01.758 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:01.754+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:01.758 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:01.754+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:01.759 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:01.754+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:01.759 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:01.754+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:01.759 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:01.754+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 0x7ff350006a70 :-1 s=OPENED pgs=1103 cs=1 l=1).handle_message read tag failed 2021-03-09T10:08:01.760 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:01.754+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 0x7ff350006a70 :-1 s=OPENED pgs=1103 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:08:01.760 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:01.754+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:08:01.805 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:01.805+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:03.391 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:03.390+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:08:05.722 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:05.717+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:06.758 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:06.754+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:06.758 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:06.754+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:06.805 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:06.801+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:09.798 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:09.794+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:08:10.744 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:10.741+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:11.758 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:11.754+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:11.759 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:11.754+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:11.805 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:11.801+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:15.746 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:15.741+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:16.758 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:16.754+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:16.759 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:16.754+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:16.805 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:16.801+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:20.768 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:20.765+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:21.758 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:21.758+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:21.759 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:21.758+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:21.805 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:21.805+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:22.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:22.606+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:08:25.770 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:25.764+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:26.758 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:26.754+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:26.759 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:26.754+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:26.805 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:26.800+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:08:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:08:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:08:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,2,5,0,7,3,6,1,4] 2021-03-09T10:08:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350005aa0 addr v1:172.21.15.143:6793/0 2021-03-09T10:08:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:08:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350005ed0 addr v1:172.21.15.173:6791/0 2021-03-09T10:08:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a20 con 0x7ff3500163b0 2021-03-09T10:08:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350005ed0 2021-03-09T10:08:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350005aa0 2021-03-09T10:08:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.170+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:08:27.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.374+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:08:27.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:27.778+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:08:28.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:28.582+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:08:30.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:30.186+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:08:30.792 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:30.788+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:30.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:30.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:30.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:30.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:30.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:30.790+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 0x7ff35000fa20 :-1 s=OPENED pgs=1120 cs=1 l=1).handle_message read tag failed 2021-03-09T10:08:30.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:30.790+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 0x7ff35000fa20 :-1 s=OPENED pgs=1120 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:08:30.794 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:30.790+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:08:31.759 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:31.758+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:31.759 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:31.758+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:31.760 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:31.758+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:31.760 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:31.758+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:31.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:31.758+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 0x7ff350006a70 :-1 s=OPENED pgs=1120 cs=1 l=1).handle_message read tag failed 2021-03-09T10:08:31.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:31.758+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 0x7ff350006a70 :-1 s=OPENED pgs=1120 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:08:31.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:31.758+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:08:31.806 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:31.804+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:33.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:33.390+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:08:35.794 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:35.792+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:36.759 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:36.757+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:36.760 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:36.757+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:36.806 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:36.804+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:39.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:39.793+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:08:40.816 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:40.812+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:41.759 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:41.753+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:41.760 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:41.753+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:41.806 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:41.804+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:45.818 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:45.816+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:46.759 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:46.757+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:46.760 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:46.757+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:46.806 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:46.800+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:50.840 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:50.836+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:51.759 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:51.753+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:51.760 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:51.753+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:51.806 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:51.804+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:52.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:52.601+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:08:55.842 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:08:55.840+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:08:56.760 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:08:56.753+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:08:56.760 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:08:56.753+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:08:56.807 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:08:56.804+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:08:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:08:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:08:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:08:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ed0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,0,6,4,7,3] 2021-03-09T10:08:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350010130 addr v1:172.21.15.173:6789/0 2021-03-09T10:08:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T10:08:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001b120 addr v1:172.21.15.143:6792/0 2021-03-09T10:08:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500111f0 2021-03-09T10:08:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a20 con 0x7ff350010130 2021-03-09T10:08:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff35001b120 2021-03-09T10:08:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:08:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.169+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:08:57.376 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.373+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:08:57.377 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.373+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:08:57.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.777+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:08:57.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:57.777+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:08:58.584 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:58.581+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:08:58.585 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:08:58.581+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:09:00.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:00.185+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:09:00.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:00.185+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:09:00.864 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:00.864+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:01.760 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:01.757+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:01.760 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:01.757+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:01.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:01.757+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 legacy=0x7ff350013870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:09:01.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:01.757+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 legacy=0x7ff350013870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:01.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:01.757+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 0x7ff350013870 :-1 s=OPENED pgs=1161 cs=1 l=1).handle_message read tag failed 2021-03-09T10:09:01.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:01.757+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 0x7ff350013870 :-1 s=OPENED pgs=1161 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:09:01.762 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:01.757+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:09:01.807 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:01.804+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:03.390 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:03.389+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:09:03.391 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:03.389+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:09:05.866 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:05.863+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:06.760 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:06.757+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:06.761 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:06.757+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:06.807 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:06.807+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:09.797 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:09.793+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:09:09.798 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:09.793+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:09:10.888 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:10.887+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:11.760 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:11.756+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:11.761 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:11.756+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:11.807 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:11.807+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:15.890 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:15.887+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:16.760 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:16.756+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:16.761 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:16.756+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:16.807 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:16.807+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:20.912 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:20.911+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:21.761 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:21.756+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:21.761 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:21.756+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:21.808 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:21.807+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:22.608+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:09:22.611 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:22.608+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:09:25.915 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:25.911+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:26.761 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:26.756+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:26.761 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:26.756+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:26.808 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:26.807+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:09:27.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:09:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:09:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001b120 legacy=0x7ff350013870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350010130 legacy=0x7ff350012150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,4,7] 2021-03-09T10:09:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:09:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350014f90 addr v1:172.21.15.173:6790/0 2021-03-09T10:09:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:09:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:09:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500163b0 addr v1:172.21.15.173:6792/0 2021-03-09T10:09:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350014f90 2021-03-09T10:09:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350005aa0 2021-03-09T10:09:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a20 con 0x7ff3500163b0 2021-03-09T10:09:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff350021ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.172+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:27.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.376+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:27.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.376+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff350021ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:27.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.780+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:27.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:27.780+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff350021ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:28.585 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:28.584+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:28.586 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:28.584+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff350021ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:30.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:30.188+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:30.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:30.188+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff350021ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:30.936 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:30.935+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:31.761 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:31.756+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:31.761 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:31.756+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:31.808 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:31.807+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:31.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:31.804+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350009ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:09:31.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:31.804+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350009ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:31.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:31.804+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350009ac0 :-1 s=OPENED pgs=1160 cs=1 l=1).handle_message read tag failed 2021-03-09T10:09:31.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:31.804+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350009ac0 :-1 s=OPENED pgs=1160 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:09:31.810 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:31.804+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:09:33.394 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:33.392+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff350021ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:33.395 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:33.392+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:35.938 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:35.935+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:36.761 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:36.760+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:36.762 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:36.760+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:36.808 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:36.807+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:39.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:39.796+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff350021ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:39.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:39.800+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:40.960 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:40.959+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:41.761 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:41.760+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:41.762 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:41.760+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:41.808 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:41.807+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:45.963 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:45.959+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:46.761 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:46.760+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:46.762 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:46.760+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:46.809 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:46.807+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:50.984 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:50.982+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:51.762 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:51.759+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:51.762 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:51.759+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:51.809 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:51.806+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:52.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:52.607+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff350021ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:52.614 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:52.611+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:55.986 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:09:55.982+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:09:56.762 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:09:56.759+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:09:56.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:09:56.759+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:09:56.809 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:09:56.806+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:09:57.176 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:09:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:09:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:09:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350009ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff350021ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,1,3,2,6,7,8,5,0] 2021-03-09T10:09:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:09:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350005eb0 addr v1:172.21.15.173:6789/0 2021-03-09T10:09:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.171+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350014f90 addr v1:172.21.15.173:6790/0 2021-03-09T10:09:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350005eb0 2021-03-09T10:09:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350014f90 2021-03-09T10:09:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350005aa0 2021-03-09T10:09:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.175+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.175+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.175+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005eb0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:09:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.175+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff35000a1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:57.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.379+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:57.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.379+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005eb0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:09:57.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.379+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff35000a1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:57.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.783+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:57.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.783+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005eb0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:09:57.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:57.783+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff35000a1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:09:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:58.587+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:09:58.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:58.587+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005eb0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:09:58.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:09:58.587+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff35000a1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:10:00.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:00.191+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:10:00.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:00.191+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005eb0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:10:00.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:00.191+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff35000a1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:10:01.008 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:01.006+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:01.762 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:01.759+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:01.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:01.759+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:01.809 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:01.806+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:03.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:03.395+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:10:03.397 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:03.395+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005eb0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:10:03.397 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:03.395+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff35000a1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:10:06.011 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:06.006+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:06.762 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:06.759+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:06.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:06.759+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:06.810 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:06.806+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:09.799+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005eb0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:10:09.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:09.799+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:10:09.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:09.799+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff35000a1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:10:11.032 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:11.030+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:11.762 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:11.759+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:11.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:11.759+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:11.810 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:11.806+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:16.034 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:16.030+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:16.763 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:16.759+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:16.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:16.759+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:16.810 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:16.806+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:21.056 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:21.054+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:21.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:21.759+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:21.763 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:21.759+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:21.810 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:21.810+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:22.607+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:10:22.610 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:22.607+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff35000a1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:10:22.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:22.615+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005eb0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:10:26.059 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:26.054+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:26.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:26.758+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:26.763 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:26.758+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:26.810 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:26.810+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:10:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:10:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:10:27.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014f90 legacy=0x7ff35000a1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005eb0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,6,5,0,7,8] 2021-03-09T10:10:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T10:10:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350015370 addr v1:172.21.15.143:6792/0 2021-03-09T10:10:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001b110 addr v1:172.21.15.173:6791/0 2021-03-09T10:10:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a20 con 0x7ff3500111f0 2021-03-09T10:10:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff35001b110 2021-03-09T10:10:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350015370 2021-03-09T10:10:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.174+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:10:27.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.378+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:10:27.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:27.782+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:10:28.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:28.586+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:10:30.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:30.190+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:10:31.080 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:31.077+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:31.081 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.078+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001b110 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:31.081 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.078+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001b110 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:31.082 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.078+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001b110 0x7ff35000fa30 :-1 s=OPENED pgs=1179 cs=1 l=1).handle_message read tag failed 2021-03-09T10:10:31.082 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.078+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001b110 0x7ff35000fa30 :-1 s=OPENED pgs=1179 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:10:31.082 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.078+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:10:31.082 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.078+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:10:31.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:31.758+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:31.764 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:31.758+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:31.764 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.758+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015370 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:31.764 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.758+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015370 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:31.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.758+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015370 0x7ff350014870 :-1 s=OPENED pgs=1216 cs=1 l=1).handle_message read tag failed 2021-03-09T10:10:31.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.758+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015370 0x7ff350014870 :-1 s=OPENED pgs=1216 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:10:31.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:31.758+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:10:31.810 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:31.805+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:33.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:33.394+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:10:36.083 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:36.077+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:36.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:36.758+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:36.764 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:36.758+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:36.810 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:36.809+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:39.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:39.798+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:10:41.108 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:41.105+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:41.763 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:41.758+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:41.764 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:41.758+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:41.810 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:41.809+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:46.111 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:46.105+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:46.764 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:46.758+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:46.764 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:46.758+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:46.811 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:46.805+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:51.132 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:51.129+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:51.764 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:51.758+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:51.764 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:51.758+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:51.811 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:51.805+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:52.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:52.610+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:10:56.134 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:10:56.129+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:10:56.764 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:10:56.762+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:10:56.765 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:10:56.762+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:10:56.811 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:10:56.809+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:10:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:10:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:10:57.177 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:10:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015370 legacy=0x7ff350014870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001b110 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,8,7] 2021-03-09T10:10:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350005eb0 addr v1:172.21.15.143:6789/0 2021-03-09T10:10:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350014f90 addr v1:172.21.15.143:6793/0 2021-03-09T10:10:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500163b0 addr v1:172.21.15.173:6792/0 2021-03-09T10:10:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350005eb0 2021-03-09T10:10:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015a20 con 0x7ff3500163b0 2021-03-09T10:10:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff350014f90 2021-03-09T10:10:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.174+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005eb0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:10:57.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.378+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005eb0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:10:57.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:57.782+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005eb0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:10:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:10:58.586+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005eb0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:00.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:00.190+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005eb0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:01.137 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:01.133+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:01.764 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:01.762+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:01.765 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:01.762+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:01.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.762+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014f90 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:01.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.762+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014f90 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:01.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.762+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014f90 0x7ff35000fa30 :-1 s=OPENED pgs=1191 cs=1 l=1).handle_message read tag failed 2021-03-09T10:11:01.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.762+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014f90 0x7ff35000fa30 :-1 s=OPENED pgs=1191 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:11:01.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.762+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:11:01.811 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:01.809+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:01.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.806+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:01.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.806+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350010140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:01.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.806+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350010140 :-1 s=OPENED pgs=1210 cs=1 l=1).handle_message read tag failed 2021-03-09T10:11:01.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.806+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350010140 :-1 s=OPENED pgs=1210 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:11:01.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:01.806+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:11:03.398 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:03.393+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005eb0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:06.156 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:06.153+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:06.764 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:06.761+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:06.765 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:06.761+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:06.812 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:06.809+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:09.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:09.801+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005eb0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:11.159 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:11.153+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:11.765 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:11.761+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:11.765 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:11.761+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:11.812 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:11.809+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:16.180 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:16.176+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:16.765 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:16.761+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:16.765 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:16.761+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:16.812 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:16.808+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:21.183 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:21.180+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:21.765 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:21.761+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:21.765 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:21.761+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:21.812 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:21.812+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:22.617 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:22.613+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005eb0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:26.204 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:26.200+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:26.765 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:26.761+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:26.766 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:26.761+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:26.812 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:26.808+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:11:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:11:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:11:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014f90 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350010140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005eb0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,2,0,7,3,8,6,5,4] 2021-03-09T10:11:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:11:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35000ae00 addr v1:172.21.15.173:6789/0 2021-03-09T10:11:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:11:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T10:11:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500111f0 2021-03-09T10:11:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35000ae00 2021-03-09T10:11:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500163b0 2021-03-09T10:11:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:11:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.173+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:11:27.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.377+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:27.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.377+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:11:27.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.377+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:11:27.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.781+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:27.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.781+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:11:27.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:27.781+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:11:28.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:28.585+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:11:28.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:28.585+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:28.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:28.585+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:11:30.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:30.189+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:30.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:30.189+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:11:30.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:30.189+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:11:31.207 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:31.204+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:31.765 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:31.761+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:31.766 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:31.761+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:31.813 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:31.808+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:33.398 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:33.393+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:11:33.399 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:33.393+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:33.399 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:33.393+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:11:36.228 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:36.224+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:36.765 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:36.761+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:36.766 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:36.761+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:36.813 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:36.812+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:39.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:39.796+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:11:39.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:39.796+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:39.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:39.796+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:11:41.231 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:41.228+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:41.766 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:41.764+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:41.766 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:41.764+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:41.813 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:41.808+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:46.252 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:46.248+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:46.766 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:46.764+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:46.766 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:46.764+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:46.813 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:46.812+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:51.255 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:51.252+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:51.766 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:51.760+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:51.767 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:51.760+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:51.813 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:51.812+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:52.614 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:52.608+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:11:52.614 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:52.608+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:11:52.615 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:52.608+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:11:56.276 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:11:56.276+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:11:56.766 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:11:56.760+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:11:56.767 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:11:56.760+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:11:56.813 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:11:56.812+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:11:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:11:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:11:57.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:11:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,6,7,4,5,3] 2021-03-09T10:11:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350010130 addr v1:172.21.15.143:6793/0 2021-03-09T10:11:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350005ab0 addr v1:172.21.15.143:6792/0 2021-03-09T10:11:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350005ec0 addr v1:172.21.15.173:6792/0 2021-03-09T10:11:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350005ab0 2021-03-09T10:11:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350005ec0 2021-03-09T10:11:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350010130 2021-03-09T10:11:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:11:57.176+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:01.279 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:01.275+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:01.767 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:01.764+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:01.767 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:01.764+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:01.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:01.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:01.768 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 0x7ff350014160 :-1 s=OPENED pgs=1228 cs=1 l=1).handle_message read tag failed 2021-03-09T10:12:01.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 0x7ff350014160 :-1 s=OPENED pgs=1228 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:12:01.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005ab0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:01.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005ab0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:01.769 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005ab0 0x7ff350014870 :-1 s=OPENED pgs=1267 cs=1 l=1).handle_message read tag failed 2021-03-09T10:12:01.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:12:01.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005ab0 0x7ff350014870 :-1 s=OPENED pgs=1267 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:12:01.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:12:01.770 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.764+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:12:01.813 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:01.811+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:01.814 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005ec0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:12:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005ec0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.812+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005ec0 0x7ff350014f80 :-1 s=OPENED pgs=1247 cs=1 l=1).handle_message read tag failed 2021-03-09T10:12:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.812+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005ec0 0x7ff350014f80 :-1 s=OPENED pgs=1247 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:12:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:01.812+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:12:06.300 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:06.299+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:06.767 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:06.764+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:06.769 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:06.764+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:06.814 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:06.811+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:11.302 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:11.299+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:11.767 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:11.764+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:11.768 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:11.764+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:11.814 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:11.811+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:16.324 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:16.323+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:16.767 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:16.763+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:16.768 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:16.763+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:16.814 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:16.811+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:21.327 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:21.323+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:21.767 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:21.763+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:21.768 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:21.763+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:21.815 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:21.811+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:26.348 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:26.347+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:26.768 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:26.767+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:26.768 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:26.767+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:26.814 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:26.811+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:27.178 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:12:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:12:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:12:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 legacy=0x7ff350014160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005ec0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005ab0 legacy=0x7ff350014870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,4,3] 2021-03-09T10:12:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350005ab0 addr v1:172.21.15.173:6791/0 2021-03-09T10:12:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350010130 addr v1:172.21.15.143:6791/0 2021-03-09T10:12:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:12:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500111f0 2021-03-09T10:12:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350010130 2021-03-09T10:12:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350005ab0 2021-03-09T10:12:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.175+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:12:27.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.379+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:27.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.379+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:12:27.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.783+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:27.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:27.783+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:12:28.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:28.587+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:28.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:28.587+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:12:30.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:30.191+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:30.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:30.191+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:12:31.351 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:31.347+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:31.352 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:31.347+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:31.352 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:31.347+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ab0 0x7ff350006a70 :-1 s=OPENED pgs=1253 cs=1 l=1).handle_message read tag failed 2021-03-09T10:12:31.352 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:31.347+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ab0 0x7ff350006a70 :-1 s=OPENED pgs=1253 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:12:31.353 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:31.347+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:12:31.353 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:31.347+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:31.768 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:31.767+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:31.768 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:31.767+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:31.815 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:31.811+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:33.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:33.395+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:33.397 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:33.395+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:12:36.372 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:36.371+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:36.768 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:36.767+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:36.769 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:36.767+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:36.815 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:36.811+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:39.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:39.799+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:39.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:39.799+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:12:41.375 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:41.371+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:41.768 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:41.767+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:41.769 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:41.767+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:41.815 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:41.815+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:46.396 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:46.394+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:46.768 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:46.767+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:46.769 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:46.767+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:46.815 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:46.814+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:51.399 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:51.394+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:51.769 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:51.766+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:51.769 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:51.766+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:51.816 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:51.814+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:52.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:52.610+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:52.613 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:52.610+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:12:56.420 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:12:56.418+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:12:56.769 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:12:56.766+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:12:56.769 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:12:56.766+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:12:56.816 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:12:56.814+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:12:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:12:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:12:57.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:12:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,0,1,3,5,8,7,6,2] 2021-03-09T10:12:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350010130 addr v1:172.21.15.143:6791/0 2021-03-09T10:12:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:12:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500157e0 addr v1:172.21.15.173:6789/0 2021-03-09T10:12:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:12:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500157e0 2021-03-09T10:12:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350010130 2021-03-09T10:12:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:12:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.174+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500157e0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:12:57.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.378+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:12:57.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.378+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:57.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.378+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500157e0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:12:57.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.782+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:12:57.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.782+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:57.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:57.782+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500157e0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:12:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:58.586+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:12:58.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:58.586+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:12:58.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:12:58.586+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500157e0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:13:00.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:00.190+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:13:00.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:00.190+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:13:00.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:00.190+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500157e0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:13:01.423 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:01.418+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:01.769 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:01.766+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:01.770 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:01.766+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:01.816 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:01.814+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:03.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:03.394+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:13:03.397 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:03.394+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:13:03.397 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:03.394+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500157e0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:13:06.444 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:06.442+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:06.769 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:06.766+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:06.770 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:06.766+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:06.816 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:06.814+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:09.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:09.798+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:13:09.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:09.798+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:13:09.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:09.798+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500157e0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:13:11.447 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:11.442+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:11.769 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:11.766+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:11.770 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:11.766+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:11.816 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:11.814+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:16.468 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:16.466+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:16.770 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:16.766+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:16.771 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:16.766+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:16.816 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:16.814+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:21.471 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:21.466+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:21.770 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:21.766+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:21.770 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:21.766+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:21.817 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:21.814+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:22.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:22.610+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:13:22.617 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:22.614+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:13:22.617 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:22.614+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500157e0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:13:26.492 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:26.490+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:26.770 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:26.765+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:26.770 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:26.765+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:26.817 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:26.814+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:13:27.179 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:13:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:13:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350010130 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500157e0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:13:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,7,5,8,2,3] 2021-03-09T10:13:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:13:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500111f0 addr v1:172.21.15.143:6792/0 2021-03-09T10:13:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500157e0 addr v1:172.21.15.173:6792/0 2021-03-09T10:13:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500163b0 addr v1:172.21.15.173:6791/0 2021-03-09T10:13:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500163b0 2021-03-09T10:13:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500111f0 2021-03-09T10:13:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500157e0 2021-03-09T10:13:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:27.173+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:31.495 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:31.489+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:31.496 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.493+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:13:31.496 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.493+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:31.496 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.493+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff35000fa20 :-1 s=OPENED pgs=1286 cs=1 l=1).handle_message read tag failed 2021-03-09T10:13:31.497 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.493+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff35000fa20 :-1 s=OPENED pgs=1286 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:13:31.497 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.493+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:13:31.770 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:31.769+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:31.771 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:31.769+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:31.771 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.769+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:31.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.769+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:31.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.769+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1320 cs=1 l=1).handle_message read tag failed 2021-03-09T10:13:31.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.769+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1320 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:13:31.772 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.769+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:13:31.817 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:31.813+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:31.819 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.813+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157e0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:31.819 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.813+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157e0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:31.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.813+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157e0 0x7ff35000ce20 :-1 s=OPENED pgs=1300 cs=1 l=1).handle_message read tag failed 2021-03-09T10:13:31.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.813+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157e0 0x7ff35000ce20 :-1 s=OPENED pgs=1300 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:13:31.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:31.813+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:13:36.516 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:36.513+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:36.770 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:36.769+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:36.771 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:36.769+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:36.817 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:36.817+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:41.519 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:41.513+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:41.771 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:41.769+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:41.771 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:41.769+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:41.817 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:41.813+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:46.540 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:46.537+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:46.771 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:46.765+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:46.771 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:46.765+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:46.818 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:46.813+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:51.542 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:51.537+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:51.771 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:51.769+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:51.771 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:51.769+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:51.818 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:51.813+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:56.545 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:13:56.541+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:13:56.771 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:13:56.765+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:13:56.772 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:13:56.765+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:13:56.818 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:13:56.813+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:13:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:13:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:13:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:13:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157e0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,2,3] 2021-03-09T10:13:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350010130 addr v1:172.21.15.143:6793/0 2021-03-09T10:13:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500053a0 addr v1:172.21.15.143:6790/0 2021-03-09T10:13:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350014870 addr v1:172.21.15.173:6790/0 2021-03-09T10:13:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500053a0 2021-03-09T10:13:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350014870 2021-03-09T10:13:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016090 con 0x7ff350010130 2021-03-09T10:13:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:13:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.173+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:13:57.380 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.377+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:13:57.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.377+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:13:57.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.781+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:13:57.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:57.781+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:13:58.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:58.585+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:13:58.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:13:58.585+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:14:00.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:00.189+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:14:00.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:00.189+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:14:01.564 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:01.561+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:01.771 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:01.765+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:01.772 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:01.765+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:01.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:01.765+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:01.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:01.765+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:01.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:01.765+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 0x7ff350014160 :-1 s=OPENED pgs=1297 cs=1 l=1).handle_message read tag failed 2021-03-09T10:14:01.773 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:01.765+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 0x7ff350014160 :-1 s=OPENED pgs=1297 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:14:01.774 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:01.765+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:14:01.818 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:01.813+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:03.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:03.392+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:14:03.397 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:03.392+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:14:06.567 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:06.561+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:06.772 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:06.768+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:06.772 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:06.768+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:06.818 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:06.813+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:09.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:09.796+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:14:09.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:09.796+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:14:11.588 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:11.585+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:11.772 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:11.768+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:11.773 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:11.768+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:11.819 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:11.817+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:16.591 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:16.589+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:16.772 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:16.764+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:16.773 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:16.764+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:16.819 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:16.813+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:21.612 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:21.608+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:21.772 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:21.768+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:21.773 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:21.768+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:21.819 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:21.816+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:22.613 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:22.612+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:14:22.614 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:22.612+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:14:26.614 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:26.612+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:26.772 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:26.768+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:26.773 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:26.768+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:26.819 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:26.816+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:27.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:14:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:14:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:14:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350010130 legacy=0x7ff350014160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,7,4,1,2,6,8,3,5] 2021-03-09T10:14:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:14:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35001a110 addr v1:172.21.15.143:6789/0 2021-03-09T10:14:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:14:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:14:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500157e0 addr v1:172.21.15.143:6791/0 2021-03-09T10:14:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001a110 2021-03-09T10:14:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500157e0 2021-03-09T10:14:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500111f0 2021-03-09T10:14:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:14:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.176+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500157e0 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:14:27.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.380+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:14:27.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.380+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500157e0 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:14:27.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.784+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:14:27.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:27.784+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500157e0 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:14:28.588 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:28.588+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:14:28.589 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:28.588+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500157e0 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:14:30.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:30.192+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:14:30.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:30.192+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500157e0 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:14:31.636 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:31.632+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:31.773 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:31.772+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:31.773 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:31.772+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:31.819 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:31.816+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:31.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:31.816+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:31.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:31.816+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:31.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:31.816+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1335 cs=1 l=1).handle_message read tag failed 2021-03-09T10:14:31.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:31.816+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1335 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:14:31.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:31.816+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:14:33.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:33.396+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:14:33.397 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:33.396+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500157e0 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:14:36.639 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:36.636+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:36.773 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:36.772+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:36.773 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:36.772+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:36.820 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:36.816+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:39.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:39.800+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:14:39.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:39.800+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500157e0 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:14:41.660 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:41.656+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:41.773 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:41.771+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:41.773 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:41.771+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:41.820 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:41.816+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:46.664 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:46.660+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:46.773 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:46.771+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:46.774 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:46.771+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:46.820 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:46.820+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:51.684 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:51.680+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:51.773 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:51.771+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:51.774 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:51.771+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:51.820 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:51.820+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:52.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:52.615+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:14:52.617 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:52.615+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500157e0 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:14:56.687 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:14:56.684+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:14:56.774 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:14:56.771+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:14:56.774 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:14:56.771+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:14:56.820 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:14:56.820+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:14:57.180 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:14:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:14:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:14:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500157e0 legacy=0x7ff350014cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,1,8,5,3,2] 2021-03-09T10:14:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T10:14:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500053a0 addr v1:172.21.15.173:6789/0 2021-03-09T10:14:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001ac00 addr v1:172.21.15.143:6793/0 2021-03-09T10:14:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500053a0 2021-03-09T10:14:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:14:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35001ac00 2021-03-09T10:14:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.175+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500053a0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:14:57.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.383+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500053a0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:14:57.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:57.787+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500053a0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:14:58.592 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:14:58.591+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500053a0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:15:00.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:00.195+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500053a0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:15:01.708 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:01.708+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:01.774 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:01.771+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:01.774 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:01.771+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:01.775 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:01.775 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:01.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff3500057b0 :-1 s=OPENED pgs=1377 cs=1 l=1).handle_message read tag failed 2021-03-09T10:15:01.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff3500057b0 :-1 s=OPENED pgs=1377 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:15:01.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001ac00 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:15:01.776 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001ac00 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:01.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001ac00 0x7ff35000fa30 :-1 s=OPENED pgs=1335 cs=1 l=1).handle_message read tag failed 2021-03-09T10:15:01.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001ac00 0x7ff35000fa30 :-1 s=OPENED pgs=1335 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:15:01.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:15:01.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:15:01.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:01.771+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:15:01.821 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:01.820+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:03.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:03.399+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500053a0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:15:06.711 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:06.708+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:06.774 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:06.771+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:06.774 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:06.771+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:06.824 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:06.820+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:09.803+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500053a0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:15:11.732 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:11.731+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:11.774 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:11.771+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:11.774 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:11.771+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:11.821 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:11.819+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:16.735 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:16.731+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:16.774 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:16.771+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:16.775 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:16.771+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:16.821 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:16.819+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:21.756 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:21.755+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:21.774 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:21.770+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:21.775 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:21.770+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:21.821 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:21.819+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:22.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:22.618+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500053a0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:15:26.759 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:26.755+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:26.774 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:26.770+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:26.777 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:26.774+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:26.821 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:26.819+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.174+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:15:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.174+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:15:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.174+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:15:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001ac00 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.174+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500053a0 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,5,2] 2021-03-09T10:15:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500163b0 addr v1:172.21.15.173:6790/0 2021-03-09T10:15:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001ac00 addr v1:172.21.15.173:6791/0 2021-03-09T10:15:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T10:15:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500111f0 2021-03-09T10:15:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500163b0 2021-03-09T10:15:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001ac00 2021-03-09T10:15:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.178+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:15:27.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.382+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:27.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.382+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:15:27.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.786+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:27.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:27.786+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:15:28.593 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:28.590+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:28.593 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:28.590+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:15:30.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:30.194+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:30.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:30.194+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:15:31.775 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:31.770+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:31.780 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:31.779+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:31.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:31.778+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001ac00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:31.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:31.778+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001ac00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:31.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:31.778+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001ac00 0x7ff350006a70 :-1 s=OPENED pgs=1356 cs=1 l=1).handle_message read tag failed 2021-03-09T10:15:31.782 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:31.778+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001ac00 0x7ff350006a70 :-1 s=OPENED pgs=1356 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:15:31.782 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:31.778+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:15:31.782 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:31.778+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:31.821 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:31.819+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:33.402 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:33.398+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:33.402 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:33.398+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:15:36.775 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:36.774+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:36.781 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:36.778+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:36.782 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:36.779+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:36.822 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:36.819+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:39.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:39.806+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:39.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:39.806+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:15:41.775 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:41.770+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:41.781 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:41.778+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:41.804 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:41.803+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:41.822 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:41.819+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:46.775 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:46.774+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:46.782 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:46.778+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:46.807 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:46.803+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:46.822 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:46.819+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:51.776 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:51.774+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:51.782 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:51.778+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:51.822 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:51.819+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:51.828 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:51.827+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:52.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:52.618+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:52.621 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:52.618+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:15:56.776 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:15:56.769+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:15:56.782 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:15:56.777+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:15:56.822 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:15:56.818+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:15:56.830 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:15:56.826+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:15:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:15:57.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:15:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:15:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001ac00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff3500057b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,8,2,7,5,4,3,0,1] 2021-03-09T10:15:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350010140 addr v1:172.21.15.143:6792/0 2021-03-09T10:15:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500157b0 addr v1:172.21.15.143:6793/0 2021-03-09T10:15:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T10:15:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016070 con 0x7ff3500111f0 2021-03-09T10:15:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff350010140 2021-03-09T10:15:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350010a00 con 0x7ff3500157b0 2021-03-09T10:15:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.177+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:57.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.381+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:57.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:57.785+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:15:58.592 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:15:58.589+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:16:00.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:00.193+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:16:01.776 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:01.769+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:01.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.773+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350010140 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:01.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.773+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350010140 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:01.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.773+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350010140 0x7ff350014160 :-1 s=OPENED pgs=1416 cs=1 l=1).handle_message read tag failed 2021-03-09T10:16:01.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.773+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350010140 0x7ff350014160 :-1 s=OPENED pgs=1416 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:16:01.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.773+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:16:01.782 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:01.777+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:01.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.777+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500157b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:01.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.777+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500157b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:01.783 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.777+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500157b0 0x7ff35001a110 :-1 s=OPENED pgs=1375 cs=1 l=1).handle_message read tag failed 2021-03-09T10:16:01.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.777+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500157b0 0x7ff35001a110 :-1 s=OPENED pgs=1375 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:16:01.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.777+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:16:01.784 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:01.777+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:16:01.823 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:01.822+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:01.852 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:01.850+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:03.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:03.397+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:16:06.776 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:06.773+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:06.782 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:06.777+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:06.823 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:06.818+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:06.854 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:06.850+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:09.801+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:16:11.776 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:11.773+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:11.783 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:11.777+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:11.823 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:11.822+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:11.876 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:11.874+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:16.777 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:16.773+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:16.783 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:16.777+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:16.823 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:16.822+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:16.878 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:16.874+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:21.777 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:21.773+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:21.783 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:21.781+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:21.823 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:21.822+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:21.900 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:21.898+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:22.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:22.617+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:16:26.777 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:26.773+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:26.783 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:26.777+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:26.824 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:26.822+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:26.902 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:26.898+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:27.181 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:16:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:16:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:16:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500157b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350010140 legacy=0x7ff350014160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,7,3,5,4,1] 2021-03-09T10:16:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350010140 addr v1:172.21.15.143:6789/0 2021-03-09T10:16:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500157b0 addr v1:172.21.15.173:6792/0 2021-03-09T10:16:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500163b0 addr v1:172.21.15.173:6790/0 2021-03-09T10:16:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350010140 2021-03-09T10:16:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016070 con 0x7ff3500163b0 2021-03-09T10:16:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001d500 con 0x7ff3500157b0 2021-03-09T10:16:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010140 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:16:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.177+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:16:27.381 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.381+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010140 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:16:27.382 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.381+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:16:27.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.785+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010140 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:16:27.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:27.785+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:16:28.590 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:28.589+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010140 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:16:28.590 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:28.589+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:16:30.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:30.193+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010140 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:16:30.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:30.193+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:16:31.777 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:31.777+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:31.783 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:31.777+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:31.824 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:31.822+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:31.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:31.821+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157b0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:31.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:31.821+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157b0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:31.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:31.821+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157b0 0x7ff3500053a0 :-1 s=OPENED pgs=1421 cs=1 l=1).handle_message read tag failed 2021-03-09T10:16:31.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:31.821+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157b0 0x7ff3500053a0 :-1 s=OPENED pgs=1421 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:16:31.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:31.821+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:16:31.924 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:31.922+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:33.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:33.396+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010140 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:16:33.401 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:33.396+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:16:36.777 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:36.776+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:36.784 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:36.780+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:36.824 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:36.822+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:36.926 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:36.922+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:39.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:39.804+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010140 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:16:39.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:39.804+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:16:41.778 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:41.776+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:41.784 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:41.780+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:41.824 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:41.822+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:41.948 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:41.946+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:46.779 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:46.776+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:46.785 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:46.780+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:46.824 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:46.821+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:46.950 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:46.945+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:51.778 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:51.776+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:51.784 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:51.784+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:51.825 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:51.821+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:51.953 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:51.949+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:52.621 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:52.620+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010140 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:16:52.621 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:52.620+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:16:56.778 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:16:56.772+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:16:56.784 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:16:56.784+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:16:56.825 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:16:56.825+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:16:56.972 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:16:56.969+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:16:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:16:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:16:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:16:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500157b0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010140 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,5,4] 2021-03-09T10:16:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350005ab0 addr v1:172.21.15.173:6789/0 2021-03-09T10:16:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350014870 addr v1:172.21.15.173:6791/0 2021-03-09T10:16:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001a110 addr v1:172.21.15.143:6791/0 2021-03-09T10:16:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071c0 con 0x7ff350005ab0 2021-03-09T10:16:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001a110 2021-03-09T10:16:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016070 con 0x7ff350014870 2021-03-09T10:16:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:16:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.180+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:16:57.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.384+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:16:57.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.384+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:16:57.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.788+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:16:57.790 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:57.788+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:16:58.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:58.592+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:16:58.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:16:58.592+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:17:00.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:00.196+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:17:00.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:00.196+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:17:01.779 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:01.776+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:01.785 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:01.784+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:01.825 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:01.821+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:01.974 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:01.969+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:01.975 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:01.972+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014870 legacy=0x7ff350014c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:01.975 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:01.972+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014870 legacy=0x7ff350014c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:01.976 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:01.972+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014870 0x7ff350014c80 :-1 s=OPENED pgs=1410 cs=1 l=1).handle_message read tag failed 2021-03-09T10:17:01.976 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:01.972+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014870 0x7ff350014c80 :-1 s=OPENED pgs=1410 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:17:01.976 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:01.972+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:17:01.976 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:01.972+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:17:03.402 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:03.400+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:17:03.403 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:03.400+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:17:06.779 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:06.776+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:06.785 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:06.784+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:06.825 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:06.825+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:06.996 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:06.993+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:09.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:09.808+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:17:09.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:09.808+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:17:11.779 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:11.775+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:11.785 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:11.783+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:11.825 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:11.821+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:11.998 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:11.993+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:16.779 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:16.775+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:16.785 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:16.783+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:16.826 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:16.821+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:17.020 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:17.017+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:21.779 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:21.775+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:21.785 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:21.783+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:21.826 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:21.821+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:22.022 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:22.017+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:22.622 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:22.619+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:17:22.623 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:22.619+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:17:26.779 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:26.775+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:26.785 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:26.783+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:26.826 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:26.825+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:27.044 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:27.041+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:17:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:17:27.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:17:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350014870 legacy=0x7ff350014c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff35000ce20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,2,6,7,0,5,8,1,4] 2021-03-09T10:17:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350014870 addr v1:172.21.15.173:6790/0 2021-03-09T10:17:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500157b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:17:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T10:17:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500157b0 2021-03-09T10:17:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071c0 con 0x7ff350014870 2021-03-09T10:17:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:17:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500157b0 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:17:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.179+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:17:27.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.383+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500157b0 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:17:27.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.383+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:17:27.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.787+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500157b0 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:17:27.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:27.787+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:17:28.592 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:28.591+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500157b0 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:17:28.593 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:28.591+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:17:30.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:30.195+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500157b0 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:17:30.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:30.195+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:17:31.779 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:31.779+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:31.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:31.779+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:17:31.780 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:31.779+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:31.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:31.779+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350014160 :-1 s=OPENED pgs=1469 cs=1 l=1).handle_message read tag failed 2021-03-09T10:17:31.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:31.779+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350014160 :-1 s=OPENED pgs=1469 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:17:31.781 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:31.779+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:17:31.786 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:31.783+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:31.826 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:31.821+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:32.047 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:32.045+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:33.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:33.399+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500157b0 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:17:33.401 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:33.399+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:17:36.780 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:36.775+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:36.786 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:36.783+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:36.826 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:36.824+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:37.068 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:37.064+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:39.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:39.803+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500157b0 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:17:39.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:39.803+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:17:41.780 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:41.779+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:41.786 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:41.783+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:41.827 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:41.820+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:42.071 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:42.064+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:46.780 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:46.779+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:46.786 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:46.783+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:46.827 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:46.820+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:47.092 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:47.088+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:51.780 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:51.778+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:51.786 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:51.786+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:51.827 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:51.824+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:52.094 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:52.088+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:52.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:52.614+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500157b0 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:17:52.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:52.618+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:17:56.780 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:17:56.778+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:17:56.787 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:17:56.782+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:17:56.827 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:17:56.824+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:17:57.116 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:17:57.112+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:17:57.182 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:17:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:17:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:17:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350014870 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500157b0 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,5,0,1,4,8] 2021-03-09T10:17:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:17:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:17:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500163b0 addr v1:172.21.15.173:6791/0 2021-03-09T10:17:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350010130 addr v1:172.21.15.143:6789/0 2021-03-09T10:17:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350010130 2021-03-09T10:17:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500163b0 2021-03-09T10:17:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071c0 con 0x7ff3500111f0 2021-03-09T10:17:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.178+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:17:57.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.382+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:17:57.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:57.786+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:17:58.592 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:17:58.590+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:18:00.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:00.194+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:18:01.781 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:01.778+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:01.787 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:01.782+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:01.827 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:01.824+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:01.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:01.826+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:01.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:01.826+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:01.829 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:01.826+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35000fa20 :-1 s=OPENED pgs=1474 cs=1 l=1).handle_message read tag failed 2021-03-09T10:18:01.829 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:01.826+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35000fa20 :-1 s=OPENED pgs=1474 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:18:01.829 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:01.826+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:18:02.119 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:02.116+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:02.119 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:02.114+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:02.120 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:02.114+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:02.120 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:02.114+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1442 cs=1 l=1).handle_message read tag failed 2021-03-09T10:18:02.120 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:02.114+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1442 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:18:02.121 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:02.114+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:18:03.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:03.398+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:18:06.781 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:06.778+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:06.787 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:06.782+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:06.827 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:06.828+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:07.140 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:07.136+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:09.802+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:18:11.781 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:11.778+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:11.787 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:11.786+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:11.828 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:11.824+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:12.142 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:12.140+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:16.781 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:16.778+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:16.787 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:16.786+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:16.828 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:16.828+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:17.164 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:17.164+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:21.781 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:21.778+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:21.788 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:21.782+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:21.828 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:21.828+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:22.166 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:22.164+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:22.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:22.606+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:18:26.781 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:26.778+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:26.788 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:26.786+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:26.828 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:26.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:27.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:18:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:18:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:18:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350010130 legacy=0x7ff35000ce10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,4,8] 2021-03-09T10:18:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350015710 addr v1:172.21.15.173:6789/0 2021-03-09T10:18:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005ab0 addr v1:172.21.15.143:6791/0 2021-03-09T10:18:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350014870 addr v1:172.21.15.143:6793/0 2021-03-09T10:18:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350015710 2021-03-09T10:18:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350005ab0 2021-03-09T10:18:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500071c0 con 0x7ff350014870 2021-03-09T10:18:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:18:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.178+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015710 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:18:27.191 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:27.187+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:27.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.382+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:18:27.385 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.382+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015710 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:18:27.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.786+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:18:27.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:27.786+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015710 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:18:28.592 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:28.589+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:18:28.593 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:28.589+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015710 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:18:30.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:30.193+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:18:30.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:30.193+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015710 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:18:31.782 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:31.777+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:31.788 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:31.785+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:31.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:31.785+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014870 legacy=0x7ff350014c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:18:31.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:31.785+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014870 legacy=0x7ff350014c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:31.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:31.785+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014870 0x7ff350014c80 :-1 s=OPENED pgs=1460 cs=1 l=1).handle_message read tag failed 2021-03-09T10:18:31.789 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:31.785+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014870 0x7ff350014c80 :-1 s=OPENED pgs=1460 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:18:31.790 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:31.785+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:18:31.828 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:31.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:32.190 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:32.187+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:33.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:33.397+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:18:33.401 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:33.397+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015710 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:18:36.782 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:36.777+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:36.788 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:36.785+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:36.828 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:36.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:37.212 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:37.211+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:39.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:39.805+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:18:39.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:39.805+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015710 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:18:41.782 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:41.777+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:41.788 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:41.785+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:41.829 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:41.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:42.214 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:42.211+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:46.782 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:46.777+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:46.789 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:46.785+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:46.829 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:46.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:47.240 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:47.239+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:51.782 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:51.781+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:51.789 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:51.785+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:51.829 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:51.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:52.242 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:52.239+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:52.622 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:52.617+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:18:52.622 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:52.617+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015710 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:18:56.782 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:18:56.777+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:18:56.789 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:18:56.785+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:18:56.829 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:18:56.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:18:57.183 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:18:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:18:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:18:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014870 legacy=0x7ff350014c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005ab0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015710 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:18:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,2,7,6,8,0,5,4,1] 2021-03-09T10:18:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:18:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:18:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:18:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35000ae00 addr v1:172.21.15.173:6792/0 2021-03-09T10:18:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500163b0 2021-03-09T10:18:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500111f0 2021-03-09T10:18:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35000ae00 2021-03-09T10:18:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:18:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.181+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:18:57.264 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:18:57.263+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:18:57.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.385+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:18:57.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.385+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:18:57.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.789+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:18:57.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:57.789+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:18:58.593 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:58.593+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:18:58.594 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:18:58.593+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:19:00.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:00.197+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:19:00.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:00.197+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:19:01.783 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:01.781+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:01.789 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:01.785+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:01.829 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:01.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:01.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:01.829+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:19:01.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:01.829+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:01.829+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ae00 0x7ff350003380 :-1 s=OPENED pgs=1500 cs=1 l=1).handle_message read tag failed 2021-03-09T10:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:01.829+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ae00 0x7ff350003380 :-1 s=OPENED pgs=1500 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:19:01.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:01.829+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:19:02.266 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:02.263+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:03.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:03.401+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:19:03.405 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:03.401+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:19:06.783 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:06.780+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:06.789 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:06.788+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:06.830 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:06.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:07.288 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:07.287+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:09.806 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:09.804+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:19:09.810 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:09.804+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:19:11.783 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:11.780+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:11.789 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:11.788+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:11.830 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:11.827+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:12.290 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:12.287+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:16.783 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:16.776+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:16.789 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:16.784+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:16.830 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:16.830+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:17.312 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:17.310+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:21.784 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:21.776+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:21.790 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:21.784+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:21.830 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:21.830+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:22.314 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:22.310+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:22.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:22.608+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:19:22.623 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:22.616+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:19:26.784 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:26.776+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:26.790 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:26.784+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:26.830 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:26.826+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:19:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:19:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:19:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ae00 legacy=0x7ff350003380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,5,6,8,0,4] 2021-03-09T10:19:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350004aa0 addr v1:172.21.15.173:6789/0 2021-03-09T10:19:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350015170 addr v1:172.21.15.173:6791/0 2021-03-09T10:19:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001a110 addr v1:172.21.15.143:6792/0 2021-03-09T10:19:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350004aa0 2021-03-09T10:19:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350015170 2021-03-09T10:19:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001a110 2021-03-09T10:19:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.180+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350004aa0 legacy=0x7ff350004e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:19:27.340 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:27.338+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:27.341 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.336+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:27.341 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.336+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:27.341 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.336+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 0x7ff350005590 :-1 s=OPENED pgs=1495 cs=1 l=1).handle_message read tag failed 2021-03-09T10:19:27.341 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.336+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 0x7ff350005590 :-1 s=OPENED pgs=1495 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:19:27.342 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.336+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:19:27.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.384+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350004aa0 legacy=0x7ff350004e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:19:27.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:27.788+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350004aa0 legacy=0x7ff350004e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:19:28.593 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:28.592+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350004aa0 legacy=0x7ff350004e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:19:30.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:30.196+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350004aa0 legacy=0x7ff350004e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:19:31.784 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:31.780+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:31.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:31.780+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:19:31.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:31.780+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:31.785 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:31.780+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a110 0x7ff35001a520 :-1 s=OPENED pgs=1532 cs=1 l=1).handle_message read tag failed 2021-03-09T10:19:31.786 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:31.780+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a110 0x7ff35001a520 :-1 s=OPENED pgs=1532 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:19:31.786 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:31.780+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:19:31.786 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:31.780+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:19:31.790 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:31.788+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:31.830 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:31.830+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:32.342 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:32.338+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:33.402 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:33.400+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350004aa0 legacy=0x7ff350004e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:19:36.784 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:36.784+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:36.790 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:36.788+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:36.831 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:36.826+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:37.364 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:37.362+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:39.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:39.808+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350004aa0 legacy=0x7ff350004e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:19:41.784 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:41.784+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:41.790 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:41.788+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:41.831 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:41.830+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:42.366 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:42.362+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:46.784 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:46.783+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:46.790 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:46.787+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:46.831 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:46.830+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:47.388 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:47.386+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:51.784 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:51.783+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:51.791 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:51.787+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:51.831 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:51.830+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:52.408 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:52.406+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:52.619 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:52.615+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350004aa0 legacy=0x7ff350004e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:19:56.785 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:19:56.783+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:19:56.791 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:19:56.787+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:19:56.832 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:19:56.830+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:19:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:19:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:19:57.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:19:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 legacy=0x7ff350005590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350004aa0 legacy=0x7ff350004e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,4,0] 2021-03-09T10:19:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350015170 addr v1:172.21.15.143:6793/0 2021-03-09T10:19:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001a110 addr v1:172.21.15.143:6791/0 2021-03-09T10:19:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:19:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500163b0 2021-03-09T10:19:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001a110 2021-03-09T10:19:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350015170 2021-03-09T10:19:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:19:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.179+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:19:57.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.383+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:19:57.384 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.383+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:19:57.410 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:19:57.406+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:19:57.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.787+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:19:57.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:57.787+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:19:58.592 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:58.591+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:19:58.592 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:19:58.591+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:00.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:00.195+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:20:00.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:00.195+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:01.785 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:01.783+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:01.791 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:01.787+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:01.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:01.787+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:01.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:01.787+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:01.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:01.787+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 0x7ff350006a70 :-1 s=OPENED pgs=1507 cs=1 l=1).handle_message read tag failed 2021-03-09T10:20:01.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:01.787+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 0x7ff350006a70 :-1 s=OPENED pgs=1507 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:20:01.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:01.787+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:20:01.832 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:01.830+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:02.432 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:02.430+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:03.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:03.399+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:20:03.401 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:03.399+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:06.785 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:06.783+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:06.791 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:06.787+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:06.832 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:06.830+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:07.434 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:07.430+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:09.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:09.803+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:20:09.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:09.803+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:11.785 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:11.783+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:11.791 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:11.787+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:11.832 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:11.829+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:12.456 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:12.453+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:16.786 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:16.783+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:16.792 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:16.791+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:16.832 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:16.829+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:17.458 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:17.453+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:21.786 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:21.783+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:21.792 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:21.791+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:21.832 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:21.829+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:22.480 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:22.477+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:22.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:22.619+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:20:22.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:22.619+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:26.786 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:26.782+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:26.792 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:26.790+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:26.832 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:26.833+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:20:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:20:27.184 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:20:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff350005590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,7,0,5,2,1,4,3,8] 2021-03-09T10:20:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35000ae00 addr v1:172.21.15.143:6792/0 2021-03-09T10:20:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:20:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:20:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500163b0 2021-03-09T10:20:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35000ae00 2021-03-09T10:20:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:20:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.182+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:27.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.386+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:27.482 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:27.477+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:27.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:27.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:28.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:28.594+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:30.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:30.198+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:31.786 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:31.782+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:31.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.782+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae00 legacy=0x7ff350004390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:31.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.782+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae00 legacy=0x7ff350004390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:31.787 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.782+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae00 0x7ff350004390 :-1 s=OPENED pgs=1574 cs=1 l=1).handle_message read tag failed 2021-03-09T10:20:31.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.782+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae00 0x7ff350004390 :-1 s=OPENED pgs=1574 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:20:31.788 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.782+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:20:31.792 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:31.790+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:31.833 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:31.833+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:31.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.830+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:31.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.830+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:31.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.830+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35001a520 :-1 s=OPENED pgs=1546 cs=1 l=1).handle_message read tag failed 2021-03-09T10:20:31.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.830+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35001a520 :-1 s=OPENED pgs=1546 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:20:31.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.830+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:20:31.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:31.830+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:20:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:32.294+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 legacy=0x7ff354032750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:20:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:32.294+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 legacy=0x7ff354032750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:32.294+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 0x7ff354032750 :-1 s=OPENED pgs=56 cs=1 l=1).handle_message read tag failed 2021-03-09T10:20:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:32.294+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 0x7ff354032750 :-1 s=OPENED pgs=56 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:20:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:32.294+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 legacy=0x7ff354032750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:32.294+0000 7ff372ffd700 10 monclient(hunting): get_auth_request con 0x7ff354043c70 auth_method 0 2021-03-09T10:20:32.503 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:32.501+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:33.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:33.402+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:36.786 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:36.782+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:36.792 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:36.790+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:36.833 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:36.829+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:37.507 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:37.501+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:39.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:39.810+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:41.786 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:41.786+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:41.793 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:41.790+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:41.833 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:41.829+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:42.528 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:42.525+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:46.787 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:46.786+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:46.793 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:46.790+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:46.833 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:46.829+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:47.532 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:47.525+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:51.787 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:51.786+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:51.793 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:51.790+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:51.833 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:51.829+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:52.552 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:52.549+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:52.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:52.622+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:20:56.787 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:20:56.782+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:20:56.793 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:20:56.790+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:20:56.834 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:20:56.833+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:20:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:20:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:20:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:20:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001a520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ae00 legacy=0x7ff350004390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,8,2,1,5,3] 2021-03-09T10:20:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35000ae00 addr v1:172.21.15.143:6791/0 2021-03-09T10:20:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500111f0 addr v1:172.21.15.143:6793/0 2021-03-09T10:20:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001cbc0 addr v1:172.21.15.143:6790/0 2021-03-09T10:20:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cbc0 2021-03-09T10:20:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.178+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35000ae00 2021-03-09T10:20:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015580 con 0x7ff3500111f0 2021-03-09T10:20:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.182+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.182+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:20:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.182+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cbc0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:20:57.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.386+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:20:57.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.386+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cbc0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:20:57.554 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:20:57.549+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:20:57.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:20:57.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:57.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cbc0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:20:58.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:58.594+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:20:58.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:20:58.594+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cbc0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:21:00.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:00.198+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:21:00.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:00.198+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cbc0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:21:01.787 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:01.786+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:01.793 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:01.790+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:01.794 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:01.790+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff35001c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:01.794 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:01.790+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff35001c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:01.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:01.790+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff35001c4b0 :-1 s=OPENED pgs=1542 cs=1 l=1).handle_message read tag failed 2021-03-09T10:21:01.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:01.790+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff35001c4b0 :-1 s=OPENED pgs=1542 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:21:01.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:01.790+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:21:01.834 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:01.828+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:02.576 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:02.572+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:03.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:03.401+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:21:03.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:03.401+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cbc0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:21:06.787 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:06.781+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:06.793 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:06.793+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:06.834 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:06.832+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:07.578 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:07.572+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:09.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:09.805+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:21:09.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:09.805+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cbc0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:21:11.787 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:11.781+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:11.793 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:11.793+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:11.834 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:11.832+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:12.599 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:12.596+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:16.788 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:16.785+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:16.794 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:16.789+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:16.834 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:16.828+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:17.602 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:17.596+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:21.788 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:21.785+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:21.794 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:21.793+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:21.835 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:21.832+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:22.623 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:22.620+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:22.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:22.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:21:22.625 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:22.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cbc0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:21:26.788 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:26.785+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:26.794 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:26.789+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:26.835 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:26.832+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:21:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:21:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:21:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff35001c4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cbc0 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,3,5] 2021-03-09T10:21:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350005ab0 addr v1:172.21.15.173:6789/0 2021-03-09T10:21:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35001cfd0 addr v1:172.21.15.173:6790/0 2021-03-09T10:21:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350015170 addr v1:172.21.15.173:6791/0 2021-03-09T10:21:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350005ab0 2021-03-09T10:21:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cfd0 2021-03-09T10:21:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350015170 2021-03-09T10:21:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:21:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.181+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:21:27.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.385+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:21:27.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.385+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:21:27.626 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:27.624+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.625+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 legacy=0x7ff350022ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:21:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.625+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 legacy=0x7ff350022ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:27.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.625+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 0x7ff350022ae0 :-1 s=OPENED pgs=1553 cs=1 l=1).handle_message read tag failed 2021-03-09T10:21:27.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.625+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 0x7ff350022ae0 :-1 s=OPENED pgs=1553 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:21:27.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.625+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:21:27.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.625+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:21:27.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.789+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:21:27.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:27.789+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:21:28.593 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:28.593+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:21:28.594 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:28.593+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:21:30.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:30.197+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:21:30.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:30.197+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:21:31.788 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:31.785+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:31.794 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:31.789+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:31.835 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:31.832+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:32.647 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:32.644+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:33.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:33.401+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:21:33.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:33.401+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:21:36.789 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:36.785+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:36.794 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:36.789+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:36.835 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:36.836+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:37.650 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:37.648+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:39.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:39.805+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:21:39.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:39.805+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:21:41.789 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:41.785+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:41.795 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:41.789+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:41.835 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:41.836+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:42.671 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:42.672+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:46.789 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:46.788+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:46.795 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:46.792+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:46.835 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:46.836+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:47.674 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:47.672+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:51.789 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:51.788+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:51.795 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:51.792+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:51.835 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:51.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:52.618 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:52.616+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:21:52.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:52.620+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:21:52.696 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:52.695+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:56.789 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:21:56.784+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:21:56.795 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:21:56.788+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:21:56.836 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:21:56.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:21:57.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:21:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:21:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:21:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015170 legacy=0x7ff350022ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:57.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350005ab0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,0,8,3,6,5,4,7,1] 2021-03-09T10:21:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff350015170 addr v1:172.21.15.143:6790/0 2021-03-09T10:21:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:21:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35000ae00 addr v1:172.21.15.143:6793/0 2021-03-09T10:21:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:21:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350015170 2021-03-09T10:21:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35000ae00 2021-03-09T10:21:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:21:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.184+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:21:57.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.388+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:21:57.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.388+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:21:57.698 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:21:57.695+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:21:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.792+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:21:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:57.792+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:21:58.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:58.596+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:21:58.598 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:21:58.596+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:22:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:00.200+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:22:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:00.200+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:22:01.789 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:01.784+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:01.795 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:01.788+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:01.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:01.788+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:01.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:01.788+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:01.797 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:01.788+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 0x7ff350005390 :-1 s=OPENED pgs=1570 cs=1 l=1).handle_message read tag failed 2021-03-09T10:22:01.797 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:01.788+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 0x7ff350005390 :-1 s=OPENED pgs=1570 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:22:01.797 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:01.788+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:22:01.836 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:01.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:02.719 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:02.719+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:03.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:03.404+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:22:03.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:03.404+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:22:06.790 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:06.788+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:06.796 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:06.792+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:06.836 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:06.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:07.722 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:07.719+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:09.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:09.812+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:22:09.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:09.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:22:11.790 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:11.788+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:11.796 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:11.796+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:11.836 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:11.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:12.743 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:12.743+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:16.790 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:16.788+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:16.796 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:16.796+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:16.836 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:16.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:17.746 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:17.743+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:21.791 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:21.787+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:21.796 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:21.795+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:21.837 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:21.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:22.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:22.627+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:22:22.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:22.627+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:22:22.767 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:22.767+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:26.790 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:26.787+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:26.796 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:26.795+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:26.837 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:26.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:27.185 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:22:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:22:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:22:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,7,1,5,4,6] 2021-03-09T10:22:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:22:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35001cfd0 addr v1:172.21.15.173:6790/0 2021-03-09T10:22:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500163b0 addr v1:172.21.15.173:6792/0 2021-03-09T10:22:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35000ae00 addr v1:172.21.15.173:6789/0 2021-03-09T10:22:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35000ae00 2021-03-09T10:22:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001cfd0 2021-03-09T10:22:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500163b0 2021-03-09T10:22:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:22:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.183+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:22:27.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.387+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:22:27.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.387+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:22:27.770 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:27.767+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:27.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.791+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:22:27.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:27.791+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:22:28.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:28.595+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:22:28.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:28.595+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:22:30.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:30.199+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:22:30.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:30.199+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:22:31.791 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:31.787+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:31.796 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:31.795+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:31.837 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:31.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:31.837 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:31.835+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:31.838 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:31.835+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:31.838 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:31.835+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1603 cs=1 l=1).handle_message read tag failed 2021-03-09T10:22:31.838 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:31.835+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1603 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:22:31.838 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:31.835+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:22:32.791 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:32.791+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:33.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:33.403+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:22:33.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:33.403+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:22:36.791 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:36.787+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:36.797 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:36.795+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:36.837 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:36.835+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:37.794 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:37.791+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:39.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:39.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:22:39.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:39.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:22:41.791 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:41.791+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:41.797 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:41.795+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:41.837 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:41.834+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:42.815 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:42.814+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:46.791 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:46.787+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:46.797 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:46.795+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:46.837 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:46.834+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:47.818 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:47.814+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:51.791 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:51.787+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:51.797 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:51.795+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:51.838 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:51.834+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:52.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:52.623+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:22:52.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:52.623+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:22:52.820 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:52.818+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:56.791 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:22:56.787+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:22:56.797 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:22:56.795+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:22:56.838 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:22:56.838+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:22:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:22:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:22:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:22:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ae00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,4,5] 2021-03-09T10:22:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500111f0 addr v1:172.21.15.143:6792/0 2021-03-09T10:22:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:22:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001cba0 addr v1:172.21.15.173:6791/0 2021-03-09T10:22:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350005aa0 2021-03-09T10:22:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35001cba0 2021-03-09T10:22:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:22:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.183+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:22:57.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.387+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:22:57.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.791+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:22:57.839 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:22:57.838+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:22:57.840 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.835+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:57.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.835+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:57.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.835+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 0x7ff35001fac0 :-1 s=OPENED pgs=1596 cs=1 l=1).handle_message read tag failed 2021-03-09T10:22:57.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.835+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 0x7ff35001fac0 :-1 s=OPENED pgs=1596 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:22:57.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:57.839+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:22:58.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:22:58.595+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:23:00.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:00.199+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:23:01.792 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:01.786+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:01.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:01.786+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:01.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:01.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:01.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:01.790+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff350005390 :-1 s=OPENED pgs=1647 cs=1 l=1).handle_message read tag failed 2021-03-09T10:23:01.793 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:01.790+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff350005390 :-1 s=OPENED pgs=1647 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:23:01.794 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:01.790+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:23:01.794 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:01.790+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:23:01.798 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:01.794+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:01.838 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:01.838+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:02.842 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:02.838+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:03.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:03.402+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:23:06.792 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:06.790+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:06.798 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:06.794+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:06.838 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:06.834+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:07.863 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:07.862+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:09.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:09.810+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:23:11.792 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:11.790+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:11.798 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:11.794+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:11.838 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:11.838+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:12.866 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:12.862+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:16.792 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:16.790+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:16.798 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:16.794+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:16.838 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:16.838+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:17.887 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:17.886+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:21.792 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:21.790+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:21.798 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:21.794+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:21.839 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:21.838+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:22.623 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:22.622+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:23:22.890 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:22.886+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:26.793 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:26.790+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:26.961 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:26.838+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:26.961 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:26.794+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:23:27.186 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:23:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:23:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,7,0,8,6,3,5,4,2] 2021-03-09T10:23:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001cba0 addr v1:172.21.15.173:6789/0 2021-03-09T10:23:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:23:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:23:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500163b0 2021-03-09T10:23:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cba0 2021-03-09T10:23:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500111f0 2021-03-09T10:23:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:23:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.182+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:23:27.386 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.386+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:23:27.387 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.386+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:23:27.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:23:27.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:27.790+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:23:27.911 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:27.910+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:28.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:28.594+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:23:28.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:28.594+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:23:30.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:30.198+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:23:30.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:30.198+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:23:31.793 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:31.790+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:31.799 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:31.794+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:31.839 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:31.838+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:31.840 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:31.834+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:31.840 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:31.834+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:31.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:31.834+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35001a820 :-1 s=OPENED pgs=1631 cs=1 l=1).handle_message read tag failed 2021-03-09T10:23:31.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:31.834+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35001a820 :-1 s=OPENED pgs=1631 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:23:31.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:31.834+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:23:32.914 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:32.910+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:33.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:33.402+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:23:33.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:33.402+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:23:36.793 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:36.790+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:36.799 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:36.794+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:36.839 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:36.837+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:37.935 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:37.933+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:39.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:39.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:23:39.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:39.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:23:41.793 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:41.789+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:41.799 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:41.797+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:41.839 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:41.837+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:42.938 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:42.933+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:46.794 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:46.793+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:46.804 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:46.793+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:46.840 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:46.837+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:47.959 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:47.957+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:51.793 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:51.793+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:51.799 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:51.793+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:51.840 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:51.837+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:52.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:52.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:23:52.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:52.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:23:52.962 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:52.957+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:56.794 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:23:56.789+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:23:56.799 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:23:56.797+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:23:56.840 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:23:56.841+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:23:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:23:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:23:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:23:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,5,4,6,2,3] 2021-03-09T10:23:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35000ae00 addr v1:172.21.15.143:6793/0 2021-03-09T10:23:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001cfd0 addr v1:172.21.15.173:6791/0 2021-03-09T10:23:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:23:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000d230 con 0x7ff350005aa0 2021-03-09T10:23:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cfd0 2021-03-09T10:23:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35000ae00 2021-03-09T10:23:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.185+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:23:57.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.389+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:23:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.793+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:23:57.987 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:23:57.985+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:23:57.989 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.981+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:57.989 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.981+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:57.990 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.981+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 0x7ff35001fac0 :-1 s=OPENED pgs=1631 cs=1 l=1).handle_message read tag failed 2021-03-09T10:23:57.990 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.981+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 0x7ff35001fac0 :-1 s=OPENED pgs=1631 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:23:57.991 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:57.981+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:23:58.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:23:58.597+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:24:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:00.201+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:24:01.794 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:01.793+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:01.800 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:01.797+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:01.797+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:01.797+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:01.797+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 0x7ff350006a70 :-1 s=OPENED pgs=1634 cs=1 l=1).handle_message read tag failed 2021-03-09T10:24:01.801 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:01.797+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 0x7ff350006a70 :-1 s=OPENED pgs=1634 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:24:01.802 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:01.797+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:24:01.802 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:01.797+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:24:01.840 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:01.841+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:02.990 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:02.985+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:03.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:03.405+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:24:06.794 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:06.789+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:06.800 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:06.797+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:06.840 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:06.841+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:08.011 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:08.009+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:09.814 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:09.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:24:11.794 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:11.793+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:11.800 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:11.797+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:11.840 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:11.837+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:13.014 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:13.009+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:16.794 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:16.793+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:16.800 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:16.797+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:16.841 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:16.841+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:18.035 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:18.033+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:21.795 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:21.792+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:21.801 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:21.800+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:21.841 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:21.837+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:22.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:22.620+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:24:23.037 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:23.033+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:26.795 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:26.788+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:26.801 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:26.796+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:26.841 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:26.836+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:24:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:24:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:24:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35000ae00 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,3,2] 2021-03-09T10:24:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001cfd0 addr v1:172.21.15.143:6792/0 2021-03-09T10:24:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35000ae00 addr v1:172.21.15.173:6790/0 2021-03-09T10:24:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:24:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500163b0 2021-03-09T10:24:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000d230 con 0x7ff35000ae00 2021-03-09T10:24:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001cfd0 2021-03-09T10:24:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.184+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:24:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.388+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:24:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.388+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:27.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.792+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:24:27.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:27.792+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:28.059 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:28.056+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:28.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:28.596+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:24:28.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:28.596+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:30.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:30.200+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:24:30.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:30.200+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:31.795 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:31.792+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:31.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:31.792+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfd0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:31.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:31.792+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfd0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:31.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:31.792+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfd0 0x7ff350006a70 :-1 s=OPENED pgs=1695 cs=1 l=1).handle_message read tag failed 2021-03-09T10:24:31.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:31.792+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfd0 0x7ff350006a70 :-1 s=OPENED pgs=1695 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:24:31.797 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:31.792+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:24:31.801 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:31.796+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:31.842 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:31.840+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:33.062 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:33.060+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:33.405 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:33.404+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:24:33.407 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:33.404+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:36.795 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:36.792+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:36.801 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:36.796+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:36.842 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:36.840+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:38.083 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:38.080+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:39.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:39.808+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:24:39.814 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:39.808+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:41.795 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:41.792+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:41.801 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:41.796+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:41.842 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:41.836+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:43.086 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:43.080+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:46.796 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:46.796+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:46.802 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:46.800+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:46.842 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:46.840+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:48.107 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:48.104+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:51.796 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:51.796+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:51.802 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:51.800+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:51.842 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:51.840+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:52.625 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:52.624+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:24:52.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:52.624+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:53.110 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:53.108+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:56.796 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:24:56.796+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:24:56.802 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:24:56.800+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:24:56.842 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:24:56.840+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:24:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:24:57.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:24:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:24:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfd0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:57.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,1,3,7,4,0,2,8,6] 2021-03-09T10:24:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500111f0 addr v1:172.21.15.173:6791/0 2021-03-09T10:24:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001cba0 addr v1:172.21.15.173:6789/0 2021-03-09T10:24:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35000ae00 addr v1:172.21.15.173:6790/0 2021-03-09T10:24:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cba0 2021-03-09T10:24:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35000ae00 2021-03-09T10:24:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000d230 con 0x7ff3500111f0 2021-03-09T10:24:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:24:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.184+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:57.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.388+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:24:57.388 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.388+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:57.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.792+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:24:57.792 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:57.792+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:24:58.131 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:24:58.128+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:24:58.132 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:58.127+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:58.132 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:58.127+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:58.132 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:58.127+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=1666 cs=1 l=1).handle_message read tag failed 2021-03-09T10:24:58.133 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:58.127+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=1666 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:24:58.133 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:58.127+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:24:58.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:58.595+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:24:58.596 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:24:58.595+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:25:00.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:00.199+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:25:00.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:00.199+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:25:01.796 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:01.795+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:01.802 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:01.799+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:01.842 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:01.844+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:03.134 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:03.132+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:03.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:03.403+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:25:03.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:03.403+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:25:06.796 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:06.795+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:06.802 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:06.799+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:06.843 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:06.844+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:08.155 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:08.156+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:09.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:09.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:25:09.811 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:09.807+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:25:11.797 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:11.795+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:11.802 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:11.799+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:11.843 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:11.844+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:13.157 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:13.156+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:16.797 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:16.795+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:16.803 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:16.799+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:16.843 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:16.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:18.179 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:18.179+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:21.797 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:21.795+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:21.803 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:21.799+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:21.843 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:21.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:22.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:22.623+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:25:22.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:22.623+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:25:23.181 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:23.179+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:26.797 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:26.795+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:26.803 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:26.799+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:26.843 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:26.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:27.187 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:25:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:25:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:25:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35000ae00 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:25:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,8,7,2,0,4] 2021-03-09T10:25:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:25:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500111f0 addr v1:172.21.15.143:6792/0 2021-03-09T10:25:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500163b0 addr v1:172.21.15.143:6793/0 2021-03-09T10:25:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001cfd0 addr v1:172.21.15.173:6792/0 2021-03-09T10:25:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:25:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cfd0 2021-03-09T10:25:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500163b0 2021-03-09T10:25:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:27.183+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:28.203 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:28.203+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:31.797 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:31.795+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:31.798 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.795+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:31.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.795+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:31.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.795+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=1738 cs=1 l=1).handle_message read tag failed 2021-03-09T10:25:31.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.795+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=1738 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:25:31.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.795+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:25:31.803 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:31.799+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:31.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.799+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:25:31.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.799+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:31.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.799+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1693 cs=1 l=1).handle_message read tag failed 2021-03-09T10:25:31.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.799+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1693 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:25:31.805 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.799+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:25:31.844 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:31.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:31.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.843+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfd0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:25:31.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.843+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfd0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:31.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.843+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfd0 0x7ff350005390 :-1 s=OPENED pgs=1702 cs=1 l=1).handle_message read tag failed 2021-03-09T10:25:31.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.843+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfd0 0x7ff350005390 :-1 s=OPENED pgs=1702 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:25:31.846 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:31.843+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:25:33.206 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:33.203+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:36.798 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:36.795+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:36.923 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:36.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:36.924 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:36.803+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:38.227 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:38.227+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:41.798 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:41.794+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:41.803 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:41.802+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:41.844 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:41.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:43.229 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:43.227+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:46.798 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:46.794+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:46.804 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:46.802+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:46.844 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:46.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:48.232 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:48.231+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:51.798 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:51.794+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:51.804 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:51.802+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:51.844 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:51.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:53.251 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:53.251+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:56.798 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:25:56.794+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:25:56.804 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:25:56.802+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:25:56.845 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:25:56.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:25:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:25:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:25:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:25:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfd0 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,2,0] 2021-03-09T10:25:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001cba0 addr v1:172.21.15.143:6791/0 2021-03-09T10:25:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff350005aa0 addr v1:172.21.15.143:6790/0 2021-03-09T10:25:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35000ce10 addr v1:172.21.15.143:6789/0 2021-03-09T10:25:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35000ce10 2021-03-09T10:25:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350005aa0 2021-03-09T10:25:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cba0 2021-03-09T10:25:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:25:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:25:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.186+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:25:57.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.390+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:25:57.393 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.390+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:25:57.393 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.390+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:25:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.794+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:25:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.794+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:25:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:57.794+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:25:58.253 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:25:58.251+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:25:58.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:58.598+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:25:58.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:58.598+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:25:58.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:25:58.598+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:00.202+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:26:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:00.202+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:00.202+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:01.798 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:01.794+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:01.804 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:01.802+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:01.845 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:01.843+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:03.275 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:03.274+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:03.407 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:03.406+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:26:03.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:03.406+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:03.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:03.406+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:06.799 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:06.798+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:06.805 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:06.802+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:06.845 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:06.842+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:08.277 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:08.274+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:09.814 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:09.814+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:26:09.814 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:09.814+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:09.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:09.814+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:11.799 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:11.794+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:11.805 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:11.802+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:11.845 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:11.846+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:13.299 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:13.298+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:16.799 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:16.797+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:16.805 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:16.801+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:16.845 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:16.842+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:18.301 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:18.298+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:21.799 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:21.793+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:21.805 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:21.805+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:21.846 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:21.842+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:22.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:22.621+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:26:22.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:22.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:22.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:22.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:23.323 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:23.322+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:26.799 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:26.797+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:26.805 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:26.801+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:26.846 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:26.842+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:27.188 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:26:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:26:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:26:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,8,4,7,2,3,6,5,1] 2021-03-09T10:26:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:26:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35000ce10 addr v1:172.21.15.143:6789/0 2021-03-09T10:26:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:26:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500111f0 addr v1:172.21.15.143:6793/0 2021-03-09T10:26:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500163b0 addr v1:172.21.15.143:6791/0 2021-03-09T10:26:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35000ce10 2021-03-09T10:26:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:26:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:26:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.185+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.389+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.389+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:27.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.793+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:27.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:27.793+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:28.326 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:28.322+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:28.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:28.597+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:28.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:28.597+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:30.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:30.201+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:30.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:30.201+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:31.801 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:31.797+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:31.806 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:31.801+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:31.807 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:31.801+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:31.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:31.801+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:31.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:31.801+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1732 cs=1 l=1).handle_message read tag failed 2021-03-09T10:26:31.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:31.801+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1732 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:26:31.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:31.801+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:26:31.846 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:31.846+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:33.347 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:33.346+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:33.405 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:33.405+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:33.405 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:33.405+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:36.800 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:36.797+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:36.806 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:36.801+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:36.846 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:36.846+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:38.349 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:38.346+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:39.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:39.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:39.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:39.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:41.800 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:41.797+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:41.806 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:41.805+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:41.846 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:41.846+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:43.371 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:43.370+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:46.800 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:46.797+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:46.806 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:46.801+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:46.846 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:46.846+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:48.373 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:48.369+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:51.801 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:51.797+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:51.806 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:51.805+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:51.847 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:51.845+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:52.625 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:52.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:26:52.625 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:52.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:26:53.399 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:53.397+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:56.801 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:26:56.796+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:26:56.806 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:26:56.800+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:26:56.847 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:26:56.845+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:26:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:26:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:26:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:26:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,5,1,2,6,7] 2021-03-09T10:26:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35001cfd0 addr v1:172.21.15.173:6790/0 2021-03-09T10:26:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001cba0 addr v1:172.21.15.173:6791/0 2021-03-09T10:26:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350014870 addr v1:172.21.15.173:6789/0 2021-03-09T10:26:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350014870 2021-03-09T10:26:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35001cfd0 2021-03-09T10:26:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cba0 2021-03-09T10:26:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014870 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:26:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.184+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:26:57.389 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.388+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014870 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:26:57.390 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.388+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:26:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.792+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014870 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:26:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:57.792+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:26:58.401 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:26:58.397+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:26:58.402 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:58.396+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:58.403 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:58.396+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:58.403 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:58.396+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 0x7ff350014160 :-1 s=OPENED pgs=1732 cs=1 l=1).handle_message read tag failed 2021-03-09T10:26:58.403 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:58.396+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 0x7ff350014160 :-1 s=OPENED pgs=1732 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:26:58.403 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:58.396+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:26:58.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:58.596+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014870 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:26:58.597 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:26:58.596+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:27:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:00.200+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014870 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:27:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:00.200+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:27:01.801 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:01.796+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:01.807 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:01.804+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:01.847 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:01.845+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:03.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:03.404+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014870 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:27:03.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:03.404+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:27:03.423 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:03.421+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:06.801 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:06.800+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:06.807 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:06.800+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:06.847 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:06.845+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:08.425 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:08.421+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:09.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:09.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014870 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:27:09.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:09.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:27:11.801 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:11.800+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:11.807 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:11.804+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:11.847 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:11.849+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:13.447 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:13.445+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:16.801 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:16.796+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:17.001 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:16.845+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:17.001 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:16.804+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:18.450 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:18.445+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:21.802 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:21.800+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:21.807 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:21.808+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:21.848 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:21.845+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:22.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:22.628+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:27:22.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:22.628+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014870 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:27:23.471 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:23.469+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:26.802 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:26.800+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:26.807 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:26.808+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:26.848 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:26.849+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:27:27.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:27:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:27:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001cfd0 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014870 legacy=0x7ff3500053a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,6,7] 2021-03-09T10:27:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001a110 addr v1:172.21.15.143:6790/0 2021-03-09T10:27:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:27:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001ac00 addr v1:172.21.15.143:6792/0 2021-03-09T10:27:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:27:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001a110 2021-03-09T10:27:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35001ac00 2021-03-09T10:27:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500111f0 2021-03-09T10:27:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.188+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:27:27.391 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.392+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:27:27.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:27.796+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:27:28.474 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:28.469+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:28.599 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:28.600+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:27:30.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:30.204+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:27:31.802 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:31.800+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:31.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.800+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:31.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.800+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:31.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.800+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 0x7ff350026b10 :-1 s=OPENED pgs=1804 cs=1 l=1).handle_message read tag failed 2021-03-09T10:27:31.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.800+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 0x7ff350026b10 :-1 s=OPENED pgs=1804 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:27:31.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.800+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:27:31.808 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:31.808+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:31.848 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:31.849+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:31.849 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.848+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:31.849 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.848+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:31.850 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.848+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1769 cs=1 l=1).handle_message read tag failed 2021-03-09T10:27:31.850 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.848+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1769 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:27:31.850 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:31.848+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:27:33.407 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:33.408+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:27:33.495 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:33.493+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:36.802 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:36.799+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:36.808 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:36.807+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:36.848 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:36.848+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:38.497 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:38.492+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:39.814 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:39.811+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:27:41.802 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:41.799+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:41.808 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:41.807+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:41.849 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:41.848+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:43.523 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:43.520+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:46.803 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:46.799+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:46.808 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:46.807+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:46.849 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:46.848+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:48.526 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:48.524+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:51.803 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:51.799+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:51.808 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:51.807+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:51.849 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:51.844+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:52.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:52.623+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:27:53.547 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:53.544+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:56.803 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:27:56.799+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:27:56.809 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:27:56.807+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:27:56.849 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:27:56.848+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:27:57.189 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:27:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:27:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:27:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 legacy=0x7ff350026b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a110 legacy=0x7ff35001a4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,7,5,0,8,6,2,3,1] 2021-03-09T10:27:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500163b0 addr v1:172.21.15.143:6791/0 2021-03-09T10:27:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:27:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001cfd0 addr v1:172.21.15.173:6791/0 2021-03-09T10:27:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:27:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cfd0 2021-03-09T10:27:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500111f0 2021-03-09T10:27:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.187+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:27:57.391 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.391+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:27:57.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:57.795+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:27:58.550 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:27:58.548+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:27:58.551 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:58.547+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:58.551 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:58.547+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:58.551 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:58.547+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 0x7ff350005390 :-1 s=OPENED pgs=1764 cs=1 l=1).handle_message read tag failed 2021-03-09T10:27:58.551 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:58.547+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 0x7ff350005390 :-1 s=OPENED pgs=1764 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:27:58.552 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:58.547+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:27:58.599 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:27:58.599+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:28:00.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:00.203+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:28:01.803 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:01.799+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:01.809 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:01.807+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:01.849 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:01.848+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:01.850 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:01.847+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:01.851 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:01.847+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:01.851 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:01.847+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1783 cs=1 l=1).handle_message read tag failed 2021-03-09T10:28:01.851 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:01.847+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1783 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:28:01.851 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:01.847+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:28:03.407 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:03.407+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:28:03.571 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:03.568+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:06.803 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:06.803+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:06.809 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:06.807+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:06.850 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:06.848+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:08.574 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:08.572+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:09.814 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:09.811+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:28:11.803 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:11.803+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:11.809 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:11.807+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:11.850 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:11.848+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:13.595 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:13.596+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:16.804 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:16.802+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:16.810 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:16.806+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:16.850 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:16.848+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:18.598 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:18.595+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:21.804 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:21.802+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:21.810 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:21.806+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:21.850 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:21.851+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:22.626 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:22.622+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:28:23.619 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:23.619+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:26.804 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:26.802+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:26.810 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:26.806+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:26.851 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:26.851+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:28:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:28:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:28:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cfd0 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,3,2,6,8,0] 2021-03-09T10:28:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001cba0 addr v1:172.21.15.173:6789/0 2021-03-09T10:28:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:28:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350005aa0 addr v1:172.21.15.173:6790/0 2021-03-09T10:28:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T10:28:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cba0 2021-03-09T10:28:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:28:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350005aa0 2021-03-09T10:28:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:28:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:28:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.186+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:28:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.390+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:28:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.390+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:28:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.390+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:28:27.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.794+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:28:27.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.794+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:28:27.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:27.794+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:28:28.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:28.598+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:28:28.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:28.598+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:28:28.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:28.598+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:28:28.621 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:28.619+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:30.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:30.202+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:28:30.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:30.202+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:28:30.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:30.202+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:28:31.804 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:31.802+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:31.810 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:31.806+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:31.851 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:31.851+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:33.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:33.406+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:28:33.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:33.406+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:28:33.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:33.406+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:28:33.643 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:33.643+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:36.806 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:36.802+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:36.810 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:36.806+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:36.851 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:36.851+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:38.646 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:38.643+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:39.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:39.810+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:28:39.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:39.810+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:28:39.814 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:39.810+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:28:41.806 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:41.806+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:41.810 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:41.806+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:41.851 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:41.851+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:43.648 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:43.647+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:46.807 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:46.806+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:46.811 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:46.806+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:46.851 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:46.851+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:48.667 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:48.667+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:51.807 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:51.801+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:51.811 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:51.809+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:51.852 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:51.851+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:52.622 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:52.617+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:28:52.623 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:52.617+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:28:52.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:52.621+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:28:53.669 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:53.667+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:28:56.807 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:28:56.801+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:28:56.811 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:28:56.805+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:28:56.852 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:28:56.851+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:28:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:28:57.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:28:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:28:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,6,8] 2021-03-09T10:28:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:28:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35000ce10 addr v1:172.21.15.143:6792/0 2021-03-09T10:28:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001cfb0 addr v1:172.21.15.143:6793/0 2021-03-09T10:28:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500163b0 2021-03-09T10:28:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35000ce10 2021-03-09T10:28:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001cfb0 2021-03-09T10:28:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.189+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:28:57.395 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.393+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:28:57.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:57.797+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:28:58.603 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:28:58.601+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:28:58.691 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:28:58.691+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:00.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:00.205+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:01.807 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:01.805+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:01.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.805+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ce10 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:01.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.805+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ce10 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:01.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.805+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ce10 0x7ff35000fa20 :-1 s=OPENED pgs=1851 cs=1 l=1).handle_message read tag failed 2021-03-09T10:29:01.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.805+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ce10 0x7ff35000fa20 :-1 s=OPENED pgs=1851 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:29:01.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.805+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:29:01.811 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:01.809+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:01.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 legacy=0x7ff350004390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:01.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 legacy=0x7ff350004390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:01.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.809+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 0x7ff350004390 :-1 s=OPENED pgs=1809 cs=1 l=1).handle_message read tag failed 2021-03-09T10:29:01.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.809+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 0x7ff350004390 :-1 s=OPENED pgs=1809 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:29:01.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.809+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:29:01.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:01.809+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:29:01.852 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:01.850+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:03.409 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:03.409+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:03.694 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:03.690+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:06.807 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:06.805+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:06.811 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:06.809+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:06.852 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:06.850+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:08.715 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:08.714+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:09.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:09.813+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:11.808 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:11.805+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:11.811 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:11.809+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:11.852 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:11.850+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:13.718 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:13.714+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:16.808 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:16.805+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:16.812 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:16.809+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:16.852 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:16.850+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:18.739 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:18.738+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:21.808 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:21.805+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:21.812 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:21.809+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:21.853 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:21.850+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:22.629 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:22.629+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:23.742 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:23.738+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:26.808 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:26.805+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:26.812 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:26.809+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:26.853 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:26.850+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:27.190 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:29:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:29:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:29:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 legacy=0x7ff350004390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000ce10 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,5,8,2,3,7,1,6,4] 2021-03-09T10:29:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:29:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000ce10 addr v1:172.21.15.173:6791/0 2021-03-09T10:29:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001cfb0 addr v1:172.21.15.143:6793/0 2021-03-09T10:29:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500163b0 2021-03-09T10:29:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35000ce10 2021-03-09T10:29:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cfb0 2021-03-09T10:29:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.189+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:27.395 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.393+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:27.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:27.797+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:28.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:28.600+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:28.763 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:28.762+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:28.764 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:28.760+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:28.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:28.760+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:28.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:28.760+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 0x7ff35001c490 :-1 s=OPENED pgs=1822 cs=1 l=1).handle_message read tag failed 2021-03-09T10:29:28.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:28.760+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 0x7ff35001c490 :-1 s=OPENED pgs=1822 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:29:28.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:28.760+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:29:30.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:30.204+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:31.808 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:31.804+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:31.813 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:31.808+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:31.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:31.808+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:31.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:31.808+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:31.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:31.808+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 0x7ff350005390 :-1 s=OPENED pgs=1831 cs=1 l=1).handle_message read tag failed 2021-03-09T10:29:31.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:31.808+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 0x7ff350005390 :-1 s=OPENED pgs=1831 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:29:31.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:31.808+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:29:31.855 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:31.854+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:33.411 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:33.408+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:33.766 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:33.762+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:36.808 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:36.804+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:36.812 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:36.808+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:36.853 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:36.854+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:38.787 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:38.786+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:39.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:39.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:41.809 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:41.804+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:41.812 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:41.808+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:41.854 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:41.854+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:43.790 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:43.785+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:46.809 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:46.804+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:46.813 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:46.808+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:46.854 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:46.849+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:48.811 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:48.809+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:51.809 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:51.804+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:51.813 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:51.808+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:51.854 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:51.853+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:52.625 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:52.624+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:29:53.814 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:53.809+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:29:56.809 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:29:56.808+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:29:56.813 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:29:56.812+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:29:56.854 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:29:56.853+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:29:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:29:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:29:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:29:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cfb0 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,7,4,2,6,1] 2021-03-09T10:29:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:29:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001cba0 addr v1:172.21.15.173:6792/0 2021-03-09T10:29:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:29:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:29:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350005aa0 2021-03-09T10:29:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cba0 2021-03-09T10:29:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:29:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.188+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:29:57.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.392+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:29:57.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.392+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:29:57.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.796+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:29:57.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:57.796+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:29:58.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:58.600+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:29:58.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:29:58.600+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:29:58.835 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:29:58.833+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:00.204+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:30:00.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:00.204+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:30:01.809 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:01.808+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:01.813 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:01.812+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:01.855 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:01.853+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:01.855 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:01.852+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:01.856 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:01.852+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:01.856 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:01.852+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 0x7ff350014870 :-1 s=OPENED pgs=1853 cs=1 l=1).handle_message read tag failed 2021-03-09T10:30:01.856 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:01.852+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 0x7ff350014870 :-1 s=OPENED pgs=1853 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:30:01.856 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:01.852+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:30:03.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:03.408+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:30:03.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:03.408+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:30:03.838 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:03.833+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:06.810 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:06.807+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:06.813 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:06.811+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:06.855 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:06.853+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:08.859 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:08.857+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:09.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:09.811+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:30:09.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:09.811+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:30:11.810 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:11.807+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:11.813 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:11.811+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:11.855 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:11.853+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:13.862 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:13.857+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:16.810 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:16.807+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:16.814 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:16.811+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:16.855 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:16.853+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:18.883 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:18.881+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:21.810 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:21.807+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:21.814 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:21.811+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:21.856 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:21.853+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:22.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:22.627+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:30:22.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:22.627+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:30:23.886 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:23.881+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:26.811 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:26.807+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:26.814 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:26.811+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:26.856 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:26.853+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:27.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:30:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:30:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:30:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,2,6] 2021-03-09T10:30:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35000fa30 addr v1:172.21.15.173:6789/0 2021-03-09T10:30:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:30:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001a820 addr v1:172.21.15.143:6790/0 2021-03-09T10:30:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001ac00 addr v1:172.21.15.143:6792/0 2021-03-09T10:30:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35000fa30 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001a820 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001ac00 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.187+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:30:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.391+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:30:27.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.391+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:27.795 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.795+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:30:27.796 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:27.795+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:28.599 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:28.599+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:30:28.600 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:28.599+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:28.907 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:28.904+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:30.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:30.203+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:30:30.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:30.203+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:31.811 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:31.807+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:31.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:31.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 legacy=0x7ff35002a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:31.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:31.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 legacy=0x7ff35002a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:31.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:31.807+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 0x7ff35002a240 :-1 s=OPENED pgs=1905 cs=1 l=1).handle_message read tag failed 2021-03-09T10:30:31.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:31.807+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 0x7ff35002a240 :-1 s=OPENED pgs=1905 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:30:31.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:31.807+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:30:31.814 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:31.811+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:31.856 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:31.856+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:33.407 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:33.407+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:30:33.409 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:33.407+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:33.910 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:33.908+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:36.811 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:36.811+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:36.815 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:36.811+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:36.856 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:36.856+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:38.931 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:38.928+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:39.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:39.811+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:30:39.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:39.815+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:41.811 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:41.811+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:41.815 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:41.811+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:41.856 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:41.852+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:43.934 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:43.932+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:46.811 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:46.810+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:46.815 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:46.810+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:46.856 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:46.856+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:48.956 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:48.952+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:51.812 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:51.810+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:51.815 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:51.814+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:51.857 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:51.852+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:52.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:52.618+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:30:52.629 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:52.626+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:53.958 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:53.956+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:56.812 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:30:56.810+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:30:56.815 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:30:56.814+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:30:56.857 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:30:56.852+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:30:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:30:57.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:30:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:30:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001ac00 legacy=0x7ff35002a240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350026b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,8,1,6,4,0,3,2,7] 2021-03-09T10:30:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500163b0 addr v1:172.21.15.173:6791/0 2021-03-09T10:30:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500111f0 addr v1:172.21.15.143:6793/0 2021-03-09T10:30:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35000fa30 addr v1:172.21.15.173:6789/0 2021-03-09T10:30:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b620 con 0x7ff35000fa30 2021-03-09T10:30:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500163b0 2021-03-09T10:30:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:30:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.190+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:57.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.394+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:57.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:57.798+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:58.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:58.602+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:30:58.979 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:30:58.976+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:30:58.980 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:58.978+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:58.981 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:58.978+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:58.981 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:58.978+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1875 cs=1 l=1).handle_message read tag failed 2021-03-09T10:30:58.981 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:58.978+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=1875 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:30:58.981 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:30:58.978+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:31:00.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:00.206+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:31:01.812 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:01.810+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:01.816 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:01.814+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:01.814+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:01.814+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:01.817 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:01.814+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=1886 cs=1 l=1).handle_message read tag failed 2021-03-09T10:31:01.817 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:01.814+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=1886 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:31:01.817 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:01.814+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:31:01.857 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:01.852+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:03.411 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:03.410+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:31:03.982 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:03.980+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:06.812 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:06.810+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:06.816 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:06.814+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:06.857 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:06.856+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:09.003 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:09.004+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:09.818 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:09.814+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:31:11.812 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:11.810+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:11.816 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:11.814+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:11.857 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:11.856+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:14.006 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:14.003+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:16.813 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:16.810+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:16.816 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:16.814+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:16.858 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:16.855+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:19.028 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:19.027+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:21.813 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:21.809+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:21.816 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:21.813+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:21.858 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:21.855+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:22.631 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:22.629+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:31:24.030 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:24.027+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:26.813 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:26.813+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:26.816 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:26.813+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:26.858 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:26.855+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:31:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:31:27.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:31:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000fa30 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,2,7,4,3,6] 2021-03-09T10:31:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35000ce20 addr v1:172.21.15.143:6789/0 2021-03-09T10:31:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:31:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001a820 addr v1:172.21.15.143:6790/0 2021-03-09T10:31:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350005aa0 addr v1:172.21.15.173:6792/0 2021-03-09T10:31:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35000ce20 2021-03-09T10:31:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.185+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b620 con 0x7ff35001a820 2021-03-09T10:31:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350005aa0 2021-03-09T10:31:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.189+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.189+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:31:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.189+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:31:27.395 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.393+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:31:27.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.393+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:31:27.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.797+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:31:27.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:27.797+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:31:28.603 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:28.601+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:31:28.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:28.601+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:31:29.052 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:29.051+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:30.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:30.205+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:31:30.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:30.205+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:31:31.813 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:31.813+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:31.817 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:31.813+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:31.858 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:31.855+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:31.859 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:31.853+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:31.859 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:31.853+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:31.860 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:31.853+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 0x7ff35001fac0 :-1 s=OPENED pgs=1913 cs=1 l=1).handle_message read tag failed 2021-03-09T10:31:31.860 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:31.853+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 0x7ff35001fac0 :-1 s=OPENED pgs=1913 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:31:31.860 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:31.853+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:31:33.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:33.409+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:31:33.413 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:33.409+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:31:34.054 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:34.051+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:36.814 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:36.813+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:36.817 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:36.813+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:36.858 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:36.855+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:39.075 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:39.075+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:39.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:39.813+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:31:39.818 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:39.813+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:31:41.814 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:41.809+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:41.817 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:41.813+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:41.859 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:41.859+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:44.100 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:44.099+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:46.814 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:46.809+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:46.817 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:46.813+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:46.859 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:46.859+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:49.102 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:49.099+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:51.814 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:51.813+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:51.817 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:51.813+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:51.859 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:51.859+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:52.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:52.629+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:31:52.631 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:52.629+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:31:54.123 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:54.123+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:31:56.814 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:31:56.809+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:31:56.818 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:31:56.813+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:31:56.859 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:31:56.859+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:31:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:31:57.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:31:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:31:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001a820 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,4,3] 2021-03-09T10:31:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500111f0 addr v1:172.21.15.143:6792/0 2021-03-09T10:31:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001ac00 addr v1:172.21.15.143:6791/0 2021-03-09T10:31:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350015170 addr v1:172.21.15.173:6790/0 2021-03-09T10:31:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500163b0 con 0x7ff350015170 2021-03-09T10:31:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001ac00 2021-03-09T10:31:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b620 con 0x7ff3500111f0 2021-03-09T10:31:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001ac00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:31:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.189+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:31:57.392 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.393+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001ac00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:31:57.393 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.393+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:31:57.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.797+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001ac00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:31:57.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:57.797+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:31:58.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:58.600+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001ac00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:31:58.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:31:58.600+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:31:59.126 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:31:59.122+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:00.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:00.204+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001ac00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:32:00.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:00.204+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:01.815 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:01.812+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:01.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:01.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:01.812+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1964 cs=1 l=1).handle_message read tag failed 2021-03-09T10:32:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:01.812+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=1964 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:32:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:01.812+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:32:01.818 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:01.812+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:01.859 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:01.858+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:03.409 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:03.408+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:03.409 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:03.408+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001ac00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:32:04.147 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:04.146+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:06.815 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:06.812+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:06.818 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:06.812+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:06.860 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:06.858+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:09.150 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:09.146+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:09.815 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:09.812+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001ac00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:32:09.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:09.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:11.815 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:11.808+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:11.818 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:11.812+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:11.860 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:11.858+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:14.172 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:14.170+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:16.815 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:16.812+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:16.818 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:16.812+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:16.860 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:16.858+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:19.174 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:19.170+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:21.815 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:21.812+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:21.818 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:21.812+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:21.860 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:21.858+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:22.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:22.628+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001ac00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:32:22.629 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:22.628+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:24.196 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:24.194+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:26.815 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:26.816+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:26.819 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:26.816+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:26.860 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:26.858+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:32:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:32:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:32:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001ac00 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015170 legacy=0x7ff35000fa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:32:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,3,1,5,2,4,8,7,6] 2021-03-09T10:32:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:32:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T10:32:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35001c490 addr v1:172.21.15.173:6790/0 2021-03-09T10:32:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35000ce10 addr v1:172.21.15.173:6789/0 2021-03-09T10:32:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500111f0 2021-03-09T10:32:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500163b0 con 0x7ff35000ce10 2021-03-09T10:32:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001c490 2021-03-09T10:32:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:32:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.192+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:32:27.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.396+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:32:27.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.396+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:27.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.396+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:32:27.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.800+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:32:27.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.800+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:27.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:27.800+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:32:28.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:28.604+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:32:28.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:28.604+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:28.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:28.604+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:32:29.198 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:29.194+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:30.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:30.208+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:32:30.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:30.208+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:30.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:30.208+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:32:31.816 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:31.816+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:31.819 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:31.816+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:31.861 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:31.858+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:33.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:33.412+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:32:33.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:33.412+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:33.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:33.412+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:32:34.220 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:34.218+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:36.816 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:36.815+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:36.819 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:36.815+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:36.861 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:36.858+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:39.222 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:39.218+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:39.818 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:39.815+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:39.819 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:39.815+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:32:39.819 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:39.815+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:32:41.816 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:41.815+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:41.819 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:41.819+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:41.861 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:41.862+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:44.243 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:44.241+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:46.816 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:46.815+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:46.819 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:46.819+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:46.861 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:46.861+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:49.246 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:49.241+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:51.816 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:51.815+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:51.819 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:51.819+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:51.861 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:51.857+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:52.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:52.619+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:32:52.632 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:52.631+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:32:52.632 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:52.631+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:32:54.267 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:54.265+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:56.817 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:32:56.815+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:32:56.820 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:32:56.819+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:32:56.862 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:32:56.861+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:32:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:32:57.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:32:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:32:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce10 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,8,5,7,2,6] 2021-03-09T10:32:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:32:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001cf80 addr v1:172.21.15.143:6793/0 2021-03-09T10:32:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000fa30 addr v1:172.21.15.173:6791/0 2021-03-09T10:32:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350005aa0 2021-03-09T10:32:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35000fa30 2021-03-09T10:32:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff3500163b0 con 0x7ff35001cf80 2021-03-09T10:32:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.191+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:32:57.395 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.395+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:32:57.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:57.799+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:32:58.603 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:58.603+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:32:59.270 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:32:59.265+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:32:59.271 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:59.267+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000fa30 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:59.271 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:59.267+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000fa30 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:59.272 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:59.267+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000fa30 0x7ff350014f80 :-1 s=OPENED pgs=1947 cs=1 l=1).handle_message read tag failed 2021-03-09T10:32:59.272 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:59.267+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000fa30 0x7ff350014f80 :-1 s=OPENED pgs=1947 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:32:59.272 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:32:59.267+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:33:00.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:00.207+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:33:01.817 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:01.815+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:01.820 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:01.819+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:01.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:01.819+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cf80 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:01.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:01.819+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cf80 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:01.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:01.819+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cf80 0x7ff350014870 :-1 s=OPENED pgs=1956 cs=1 l=1).handle_message read tag failed 2021-03-09T10:33:01.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:01.819+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cf80 0x7ff350014870 :-1 s=OPENED pgs=1956 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:33:01.822 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:01.819+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:33:01.822 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:01.819+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:33:01.862 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:01.857+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:03.411 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:03.411+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:33:04.291 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:04.289+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:06.817 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:06.815+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:06.820 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:06.819+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:06.862 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:06.861+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:09.294 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:09.289+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:09.818 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:09.814+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:33:11.817 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:11.814+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:11.820 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:11.818+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:11.862 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:11.861+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:14.316 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:14.313+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:16.819 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:16.814+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:16.821 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:16.818+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:16.863 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:16.857+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:19.318 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:19.313+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:21.817 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:21.814+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:21.821 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:21.818+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:21.862 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:21.861+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:22.630 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:22.630+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:33:24.339 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:24.337+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:26.818 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:26.814+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:26.821 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:26.818+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:26.863 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:26.861+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:27.193 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:33:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:33:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:33:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cf80 legacy=0x7ff350014870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000fa30 legacy=0x7ff350014f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,2,7] 2021-03-09T10:33:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35000fa30 addr v1:172.21.15.143:6792/0 2021-03-09T10:33:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001cf80 addr v1:172.21.15.143:6790/0 2021-03-09T10:33:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500163b0 addr v1:172.21.15.173:6792/0 2021-03-09T10:33:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cf80 2021-03-09T10:33:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35000fa30 2021-03-09T10:33:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b5f0 con 0x7ff3500163b0 2021-03-09T10:33:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.190+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:33:27.395 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.394+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:33:27.799 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:27.798+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:33:28.603 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:28.602+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:33:29.342 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:29.336+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:30.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:30.206+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:33:31.818 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:31.818+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:31.819 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.818+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:31.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.818+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:31.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.818+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 0x7ff350006a70 :-1 s=OPENED pgs=2011 cs=1 l=1).handle_message read tag failed 2021-03-09T10:33:31.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.818+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 0x7ff350006a70 :-1 s=OPENED pgs=2011 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:33:31.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.818+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:33:31.821 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:31.818+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:31.863 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:31.860+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:31.864 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.858+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:31.864 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.858+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.858+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350005390 :-1 s=OPENED pgs=1974 cs=1 l=1).handle_message read tag failed 2021-03-09T10:33:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.858+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350005390 :-1 s=OPENED pgs=1974 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:33:31.865 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:31.858+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:33:33.411 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:33.410+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:33:34.364 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:34.360+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:36.818 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:36.814+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:36.821 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:36.822+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:36.863 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:36.860+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:39.366 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:39.364+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:39.818 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:39.818+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:33:41.818 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:41.814+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:41.821 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:41.822+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:41.863 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:41.860+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:44.387 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:44.384+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:46.818 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:46.817+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:46.822 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:46.817+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:46.863 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:46.860+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:49.390 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:49.388+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:51.819 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:51.817+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:51.822 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:51.817+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:51.864 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:51.864+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:52.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:52.625+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:33:54.412 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:54.408+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:33:56.819 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:33:56.813+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:33:56.822 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:33:56.817+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:33:56.864 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:33:56.864+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:33:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:33:57.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:33:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:33:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,6,7,5,8,3,2,0,1] 2021-03-09T10:33:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:33:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35000fa30 addr v1:172.21.15.143:6792/0 2021-03-09T10:33:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500163b0 addr v1:172.21.15.173:6792/0 2021-03-09T10:33:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350005aa0 2021-03-09T10:33:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35000fa30 2021-03-09T10:33:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:33:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.193+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:33:57.399 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.397+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:33:57.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:57.801+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:33:58.607 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:33:58.605+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:33:59.414 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:33:59.412+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:00.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:00.209+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:34:01.819 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:01.813+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.813+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.813+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.813+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 0x7ff350006a70 :-1 s=OPENED pgs=2028 cs=1 l=1).handle_message read tag failed 2021-03-09T10:34:01.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.813+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 0x7ff350006a70 :-1 s=OPENED pgs=2028 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:34:01.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.813+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:34:01.822 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:01.817+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:01.864 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:01.860+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:01.865 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.861+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:01.865 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.861+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:01.865 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.861+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350005390 :-1 s=OPENED pgs=1988 cs=1 l=1).handle_message read tag failed 2021-03-09T10:34:01.865 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.861+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff350005390 :-1 s=OPENED pgs=1988 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:34:01.866 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:01.861+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:34:03.414 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:03.413+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:34:04.436 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:04.432+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:06.819 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:06.817+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:06.823 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:06.817+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:06.864 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:06.864+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:09.438 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:09.436+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:09.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:09.821+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:34:11.820 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:11.817+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:11.823 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:11.817+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:11.864 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:11.864+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:14.460 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:14.460+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:16.820 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:16.817+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:16.823 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:16.817+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:16.864 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:16.863+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:19.463 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:19.463+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:21.820 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:21.816+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:21.823 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:21.816+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:21.865 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:21.863+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:22.626 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:22.624+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:34:24.483 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:24.483+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:26.820 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:26.816+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:26.823 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:26.816+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:26.865 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:26.863+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:27.194 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:34:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:34:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:34:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35000fa30 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:27.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,2,5,8,1,3] 2021-03-09T10:34:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35001cba0 addr v1:172.21.15.143:6789/0 2021-03-09T10:34:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001cf80 addr v1:172.21.15.143:6790/0 2021-03-09T10:34:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000ce20 addr v1:172.21.15.173:6791/0 2021-03-09T10:34:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b5b0 con 0x7ff35001cba0 2021-03-09T10:34:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35001cf80 2021-03-09T10:34:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:34:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35000ce20 2021-03-09T10:34:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.192+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:34:27.399 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.396+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:34:27.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.396+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:34:27.803 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.800+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:34:27.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:27.800+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:34:28.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:28.604+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:34:28.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:28.604+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:34:29.486 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:29.483+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:29.487 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:29.484+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:29.487 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:29.484+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:29.487 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:29.484+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 0x7ff350014f80 :-1 s=OPENED pgs=1995 cs=1 l=1).handle_message read tag failed 2021-03-09T10:34:29.487 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:29.484+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 0x7ff350014f80 :-1 s=OPENED pgs=1995 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:34:29.488 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:29.484+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:34:30.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:30.208+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:34:30.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:30.208+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:34:31.820 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:31.816+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:31.824 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:31.820+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:31.865 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:31.863+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:33.415 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:33.412+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:34:33.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:33.412+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:34:34.508 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:34.507+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:36.821 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:36.816+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:36.824 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:36.820+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:36.865 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:36.863+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:39.510 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:39.507+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:39.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:39.816+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:34:39.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:39.816+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:34:41.821 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:41.816+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:41.824 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:41.824+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:41.866 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:41.863+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:44.512 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:44.511+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:46.821 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:46.820+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:46.824 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:46.824+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:46.866 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:46.863+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:49.531 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:49.531+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:51.821 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:51.820+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:51.824 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:51.824+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:51.866 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:51.867+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:52.621 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:52.620+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:34:52.632 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:52.632+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:34:54.534 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:54.531+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:34:56.821 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:34:56.820+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:34:56.824 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:34:56.824+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:34:56.866 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:34:56.867+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:34:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:34:57.195 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:34:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:34:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cf80 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,8,3] 2021-03-09T10:34:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:34:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500111f0 addr v1:172.21.15.173:6789/0 2021-03-09T10:34:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:34:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500163b0 addr v1:172.21.15.143:6793/0 2021-03-09T10:34:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350006a70 addr v1:172.21.15.173:6790/0 2021-03-09T10:34:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500111f0 2021-03-09T10:34:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b5b0 con 0x7ff350006a70 2021-03-09T10:34:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500163b0 2021-03-09T10:34:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:34:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.192+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350006a70 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:34:57.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.395+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:34:57.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.395+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350006a70 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:34:57.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.799+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350006a70 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:34:57.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:57.799+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:34:58.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:58.603+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350006a70 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:34:58.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:34:58.603+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:34:59.556 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:34:59.555+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:00.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:00.207+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350006a70 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:35:00.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:00.207+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:35:01.821 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:01.819+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:01.825 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:01.823+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:01.823+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:01.823+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:01.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:01.823+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35001a820 :-1 s=OPENED pgs=2016 cs=1 l=1).handle_message read tag failed 2021-03-09T10:35:01.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:01.823+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35001a820 :-1 s=OPENED pgs=2016 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:35:01.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:01.823+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:35:01.866 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:01.862+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:03.411 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:03.411+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350006a70 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:35:03.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:03.411+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:35:04.558 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:04.554+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:06.821 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:06.819+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:06.825 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:06.823+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:06.866 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:06.862+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:09.580 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:09.578+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:09.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:09.815+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350006a70 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:35:09.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:09.815+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:35:11.822 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:11.819+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:11.825 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:11.823+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:11.867 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:11.862+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:14.582 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:14.578+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:16.822 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:16.819+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:16.825 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:16.823+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:16.867 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:16.866+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:19.604 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:19.602+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:21.822 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:21.819+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:21.825 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:21.823+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:21.867 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:21.866+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:22.629 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:22.627+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350006a70 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:35:22.630 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:22.627+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:35:24.606 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:24.602+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:26.822 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:26.819+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:26.825 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:26.823+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:26.867 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:26.866+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:35:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:35:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:35:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350006a70 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,7,8,1,5,3,6,0,2] 2021-03-09T10:35:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35000fa20 addr v1:172.21.15.143:6791/0 2021-03-09T10:35:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35000ce20 addr v1:172.21.15.173:6792/0 2021-03-09T10:35:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500163b0 addr v1:172.21.15.143:6793/0 2021-03-09T10:35:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe40 con 0x7ff35000fa20 2021-03-09T10:35:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35000ce20 2021-03-09T10:35:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b5b0 con 0x7ff3500163b0 2021-03-09T10:35:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.191+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000fa20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:35:27.396 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.395+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000fa20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:35:27.800 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:27.799+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000fa20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:35:28.604 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:28.603+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000fa20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:35:29.627 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:29.626+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:30.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:30.207+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000fa20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:35:31.822 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:31.819+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:31.826 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:31.823+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:31.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.823+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:31.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.823+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:31.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.823+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35001a820 :-1 s=OPENED pgs=2035 cs=1 l=1).handle_message read tag failed 2021-03-09T10:35:31.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.823+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35001a820 :-1 s=OPENED pgs=2035 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:35:31.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.823+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:35:31.867 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:31.866+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:31.868 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.863+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.863+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.863+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce20 0x7ff350005390 :-1 s=OPENED pgs=2033 cs=1 l=1).handle_message read tag failed 2021-03-09T10:35:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.863+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce20 0x7ff350005390 :-1 s=OPENED pgs=2033 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:35:31.869 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:31.867+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:35:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:32.295+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354043c70 legacy=0x7ff3540402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:35:32.298 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:32.295+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354043c70 legacy=0x7ff3540402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:32.295+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354043c70 0x7ff3540402e0 :-1 s=OPENED pgs=71 cs=1 l=1).handle_message read tag failed 2021-03-09T10:35:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:32.295+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354043c70 0x7ff3540402e0 :-1 s=OPENED pgs=71 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:35:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:32.295+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354043c70 legacy=0x7ff3540402e0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:35:32.299 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:32.295+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354043c70 legacy=0x7ff3540402e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:32.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:32.295+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:35:32.300 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:32.295+0000 7ff372ffd700 10 monclient(hunting): get_auth_request con 0x7ff354030290 auth_method 0 2021-03-09T10:35:33.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:33.411+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000fa20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:35:34.630 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:34.626+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:36.823 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:36.818+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:36.938 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:36.866+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:36.939 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:36.822+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:39.652 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:39.650+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:39.819 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:39.814+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000fa20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:35:41.823 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:41.822+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:41.826 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:41.822+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:41.868 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:41.866+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:44.660 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:44.650+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:46.824 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:46.818+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:46.828 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:46.822+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:46.869 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:46.866+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:49.676 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:49.673+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:51.823 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:51.818+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:51.826 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:51.826+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:51.868 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:51.865+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:52.632 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:52.630+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000fa20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:35:54.678 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:54.673+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:35:56.823 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:35:56.818+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:35:56.827 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:35:56.822+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:35:56.868 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:35:56.865+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:35:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:35:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:35:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:35:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35000ce20 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000fa20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,6,2,1,3,5] 2021-03-09T10:35:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35000ce20 addr v1:172.21.15.143:6789/0 2021-03-09T10:35:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500163b0 addr v1:172.21.15.143:6792/0 2021-03-09T10:35:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T10:35:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35000ce20 2021-03-09T10:35:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe40 con 0x7ff3500111f0 2021-03-09T10:35:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:35:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:35:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.194+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:35:57.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.398+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:35:57.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.398+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:35:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.802+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:35:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:57.802+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:35:58.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:58.606+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:35:58.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:35:58.606+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:35:59.700 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:35:59.697+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:00.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:00.210+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:36:00.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:00.210+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:36:01.824 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:01.822+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:01.824 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:01.822+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:01.822+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:01.822+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=2088 cs=1 l=1).handle_message read tag failed 2021-03-09T10:36:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:01.822+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=2088 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:36:01.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:01.822+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:36:01.827 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:01.822+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:01.868 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:01.869+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:03.415 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:03.414+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:36:03.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:03.414+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:36:04.701 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:04.697+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:06.824 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:06.822+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:06.827 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:06.822+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:06.869 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:06.869+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:09.724 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:09.721+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:09.822 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:09.822+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:36:09.822 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:09.822+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:36:11.824 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:11.821+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:11.827 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:11.821+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:11.869 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:11.869+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:14.726 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:14.721+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:16.824 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:16.821+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:16.827 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:16.825+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:16.869 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:16.869+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:19.748 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:19.745+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:21.825 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:21.821+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:21.828 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:21.825+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:21.869 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:21.865+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:22.635 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:22.629+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:36:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:22.629+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:36:24.750 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:24.745+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:26.825 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:26.821+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:26.828 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:26.825+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:26.869 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:26.869+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:27.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:36:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:36:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:36:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,1,5] 2021-03-09T10:36:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350005aa0 addr v1:172.21.15.173:6790/0 2021-03-09T10:36:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350015170 addr v1:172.21.15.173:6789/0 2021-03-09T10:36:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350020ad0 addr v1:172.21.15.173:6791/0 2021-03-09T10:36:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350015170 2021-03-09T10:36:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350005aa0 2021-03-09T10:36:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe40 con 0x7ff350020ad0 2021-03-09T10:36:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.189+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.193+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:36:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.193+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:36:27.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.397+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:36:27.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.397+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:36:27.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.801+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:36:27.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:27.801+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:36:28.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:28.605+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:36:28.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:28.605+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:36:29.776 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:29.773+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:29.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:29.773+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350020ad0 legacy=0x7ff350022ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:29.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:29.773+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350020ad0 legacy=0x7ff350022ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:29.777 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:29.773+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350020ad0 0x7ff350022ec0 :-1 s=OPENED pgs=2056 cs=1 l=1).handle_message read tag failed 2021-03-09T10:36:29.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:29.773+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350020ad0 0x7ff350022ec0 :-1 s=OPENED pgs=2056 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:36:29.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:29.773+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:36:29.778 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:29.773+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:36:30.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:30.209+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:36:30.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:30.209+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:36:31.825 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:31.821+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:31.828 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:31.825+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:31.869 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:31.865+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:33.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:33.413+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:36:33.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:33.413+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:36:34.778 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:34.777+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:36.825 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:36.821+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:36.828 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:36.825+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:36.870 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:36.868+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:39.800 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:39.796+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:39.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:39.817+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:36:39.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:39.817+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:36:41.825 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:41.825+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:41.828 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:41.825+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:41.870 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:41.868+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:44.802 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:44.796+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:46.825 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:46.821+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:46.829 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:46.825+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:46.870 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:46.868+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:49.824 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:49.820+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:51.826 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:51.824+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:51.829 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:51.824+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:51.870 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:51.864+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:52.636 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:52.632+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:36:52.636 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:52.632+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:36:54.826 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:54.820+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:36:56.826 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:36:56.820+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:36:56.829 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:36:56.824+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:36:56.870 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:36:56.864+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:36:57.196 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:36:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:36:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:36:57.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350020ad0 legacy=0x7ff350022ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350015170 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,6,2,4,0,3,5,8,1] 2021-03-09T10:36:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350020ad0 addr v1:172.21.15.173:6792/0 2021-03-09T10:36:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500111f0 addr v1:172.21.15.143:6792/0 2021-03-09T10:36:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:36:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:36:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500111f0 2021-03-09T10:36:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350020ad0 2021-03-09T10:36:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.192+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:36:57.397 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.396+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:36:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:57.800+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:36:58.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:36:58.604+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:36:59.848 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:36:59.844+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:00.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:00.208+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:37:01.826 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:01.824+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:01.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.824+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:01.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.824+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:01.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.824+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=2128 cs=1 l=1).handle_message read tag failed 2021-03-09T10:37:01.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.824+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=2128 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:37:01.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.824+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:37:01.829 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:01.824+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:01.871 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:01.864+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:01.871 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.864+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350020ad0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:01.871 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.864+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350020ad0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:01.872 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.864+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350020ad0 0x7ff350006a70 :-1 s=OPENED pgs=2082 cs=1 l=1).handle_message read tag failed 2021-03-09T10:37:01.872 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.864+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350020ad0 0x7ff350006a70 :-1 s=OPENED pgs=2082 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:37:01.872 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:01.864+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:37:03.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:03.416+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:37:04.850 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:04.844+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:06.827 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:06.820+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:06.829 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:06.824+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:06.871 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:06.868+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:09.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:09.820+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:37:09.872 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:09.868+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:11.827 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:11.824+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:11.830 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:11.828+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:11.871 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:11.872+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:14.875 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:14.872+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:16.827 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:16.824+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:16.830 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:16.828+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:16.871 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:16.872+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:19.896 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:19.896+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:21.827 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:21.824+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:21.830 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:21.828+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:21.871 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:21.872+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:22.636+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:37:24.898 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:24.895+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:26.827 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:26.823+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:26.830 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:26.827+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:26.871 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:26.871+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:37:27.197 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:37:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:37:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350020ad0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,0,8,3,4,1] 2021-03-09T10:37:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:37:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000ce20 addr v1:172.21.15.173:6791/0 2021-03-09T10:37:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T10:37:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350015170 addr v1:172.21.15.143:6793/0 2021-03-09T10:37:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015580 con 0x7ff3500111f0 2021-03-09T10:37:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35000ce20 2021-03-09T10:37:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:37:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350015170 2021-03-09T10:37:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.195+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:27.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.399+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:37:27.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:27.803+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:37:28.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:28.607+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:37:29.919 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:29.919+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:29.920 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:29.919+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:29.921 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:29.919+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:29.921 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:29.919+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 0x7ff35001cba0 :-1 s=OPENED pgs=2095 cs=1 l=1).handle_message read tag failed 2021-03-09T10:37:29.921 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:29.919+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 0x7ff35001cba0 :-1 s=OPENED pgs=2095 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:37:29.921 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:29.919+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:37:30.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:30.211+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:37:31.827 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:31.823+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:31.831 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:31.827+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:31.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:31.827+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:37:31.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:31.827+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:31.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:31.827+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 0x7ff350006a70 :-1 s=OPENED pgs=2102 cs=1 l=1).handle_message read tag failed 2021-03-09T10:37:31.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:31.827+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 0x7ff350006a70 :-1 s=OPENED pgs=2102 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:37:31.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:31.827+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:37:31.872 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:31.871+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:33.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:33.415+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:37:34.922 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:34.919+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:36.828 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:36.827+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:36.831 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:36.827+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:36.872 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:36.871+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:39.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:39.819+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:37:39.924 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:39.923+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:41.830 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:41.827+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:41.833 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:41.827+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:41.872 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:41.871+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:44.944 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:44.943+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:46.828 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:46.827+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:46.831 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:46.827+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:46.872 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:46.871+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:49.946 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:49.943+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:51.828 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:51.827+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:51.831 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:51.827+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:51.872 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:51.871+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:52.635 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:52.631+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:37:54.968 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:54.967+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:37:56.828 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:37:56.827+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:37:56.831 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:37:56.831+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:37:56.873 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:37:56.871+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:37:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:37:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:37:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:37:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,3,1] 2021-03-09T10:37:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500163b0 addr v1:172.21.15.143:6791/0 2021-03-09T10:37:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350005aa0 addr v1:172.21.15.173:6790/0 2021-03-09T10:37:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001a820 addr v1:172.21.15.173:6789/0 2021-03-09T10:37:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff35001a820 2021-03-09T10:37:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015580 con 0x7ff350005aa0 2021-03-09T10:37:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500163b0 2021-03-09T10:37:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:37:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:37:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.195+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:37:57.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.399+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:37:57.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.399+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:37:57.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.399+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:37:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.803+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:37:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.803+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:37:57.805 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:57.803+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:37:58.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:58.607+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:37:58.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:58.607+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:37:58.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:37:58.607+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:37:59.970 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:37:59.967+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:00.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:00.211+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:00.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:00.211+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:38:00.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:00.211+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:38:01.829 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:01.826+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:01.832 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:01.830+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:01.873 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:01.871+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:03.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:03.414+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:03.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:03.414+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:38:03.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:03.414+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:38:04.992 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:04.991+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:06.829 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:06.826+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:06.832 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:06.830+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:06.873 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:06.871+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:09.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:09.822+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:09.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:09.822+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:38:09.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:09.822+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:38:09.994 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:09.991+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:11.829 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:11.826+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:11.832 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:11.830+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:11.873 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:11.870+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:15.015 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:15.014+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:16.829 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:16.826+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:16.832 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:16.830+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:16.873 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:16.870+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:20.018 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:20.014+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:21.829 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:21.826+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:21.832 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:21.830+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:21.873 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:21.874+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:22.628 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:22.626+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:38:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:22.634+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:22.634+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:38:25.039 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:25.038+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:26.830 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:26.826+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:26.832 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:26.830+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:26.874 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:26.870+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:38:27.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:38:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:38:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,2,4,1,3,6,5,0,7] 2021-03-09T10:38:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:38:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350015170 addr v1:172.21.15.143:6793/0 2021-03-09T10:38:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:38:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35000ce20 addr v1:172.21.15.143:6790/0 2021-03-09T10:38:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500111f0 addr v1:172.21.15.143:6791/0 2021-03-09T10:38:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35000ce20 2021-03-09T10:38:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500111f0 2021-03-09T10:38:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015580 con 0x7ff350015170 2021-03-09T10:38:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:38:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.194+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:27.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.398+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:38:27.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.398+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:27.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.802+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:27.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:27.802+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:38:28.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:28.606+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:38:28.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:28.606+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:30.041 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:30.038+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:30.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:30.210+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:30.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:30.210+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:38:31.830 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:31.826+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:31.833 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:31.830+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:31.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:31.830+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:31.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:31.830+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:31.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:31.830+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 0x7ff350006a70 :-1 s=OPENED pgs=2144 cs=1 l=1).handle_message read tag failed 2021-03-09T10:38:31.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:31.830+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 0x7ff350006a70 :-1 s=OPENED pgs=2144 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:38:31.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:31.830+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:38:31.874 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:31.870+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:33.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:33.414+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:33.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:33.414+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:38:35.064 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:35.062+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:36.830 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:36.826+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:36.833 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:36.830+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:36.874 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:36.870+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:39.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:39.817+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:38:39.822 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:39.817+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:40.072 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:40.062+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:41.830 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:41.825+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:41.833 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:41.829+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:41.875 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:41.870+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:45.088 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:45.086+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:46.830 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:46.825+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:46.833 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:46.829+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:46.875 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:46.874+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:50.090 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:50.086+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:51.831 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:51.825+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:51.834 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:51.833+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:51.875 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:51.874+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:52.632 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:52.629+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:38:52.635 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:52.629+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:38:55.112 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:38:55.110+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:38:56.831 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:38:56.825+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:38:56.834 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:38:56.829+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:38:56.875 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:38:56.874+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:38:57.198 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:38:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:38:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:38:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350015170 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,6,0,5,3,7] 2021-03-09T10:38:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500163b0 addr v1:172.21.15.173:6789/0 2021-03-09T10:38:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350005aa0 addr v1:172.21.15.143:6792/0 2021-03-09T10:38:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35001a820 addr v1:172.21.15.143:6789/0 2021-03-09T10:38:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001a820 2021-03-09T10:38:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:38:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff350005aa0 2021-03-09T10:38:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:38:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.193+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:38:57.398 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.397+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:38:57.399 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.397+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:38:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.801+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:38:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:57.801+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:38:58.605 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:58.605+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:38:58.606 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:38:58.605+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:39:00.114 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:00.109+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:00.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:00.209+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:39:00.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:00.209+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:39:01.831 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:01.825+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:01.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:01.825+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:01.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:01.825+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:01.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:01.825+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 0x7ff350019100 :-1 s=OPENED pgs=2210 cs=1 l=1).handle_message read tag failed 2021-03-09T10:39:01.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:01.825+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 0x7ff350019100 :-1 s=OPENED pgs=2210 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:39:01.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:01.825+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:39:01.834 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:01.829+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:01.876 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:01.873+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:03.414 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:03.413+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:39:03.415 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:03.413+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:39:05.136 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:05.133+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:06.831 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:06.829+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:06.834 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:06.829+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:06.875 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:06.873+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:09.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:09.817+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:39:09.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:09.817+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:39:10.138 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:10.133+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:11.831 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:11.825+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:11.834 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:11.829+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:11.876 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:11.873+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:15.163 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:15.161+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:16.832 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:16.828+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:16.834 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:16.828+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:16.876 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:16.873+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:20.166 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:20.161+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:21.832 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:21.828+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:21.835 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:21.828+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:21.876 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:21.877+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:22.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:22.624+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:39:22.627 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:22.624+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:39:25.188 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:25.185+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:26.832 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:26.828+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:26.835 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:26.828+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:26.876 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:26.877+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:39:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:39:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:39:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001a820 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,5,3] 2021-03-09T10:39:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001ac30 addr v1:172.21.15.173:6792/0 2021-03-09T10:39:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:39:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350024910 addr v1:172.21.15.173:6791/0 2021-03-09T10:39:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:39:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500111f0 2021-03-09T10:39:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350024910 2021-03-09T10:39:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001ac30 2021-03-09T10:39:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.192+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:39:27.399 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.396+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:39:27.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:27.800+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:39:28.605 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:28.604+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:39:30.190 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:30.185+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:30.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:30.184+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350024910 legacy=0x7ff350026d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:30.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:30.184+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350024910 legacy=0x7ff350026d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:30.191 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:30.184+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350024910 0x7ff350026d00 :-1 s=OPENED pgs=2178 cs=1 l=1).handle_message read tag failed 2021-03-09T10:39:30.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:30.184+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350024910 0x7ff350026d00 :-1 s=OPENED pgs=2178 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:39:30.192 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:30.184+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:39:30.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:30.208+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:39:31.832 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:31.828+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:31.835 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:31.828+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:31.877 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:31.873+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:31.878 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:31.872+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac30 legacy=0x7ff3500231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:31.878 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:31.872+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac30 legacy=0x7ff3500231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:31.878 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:31.872+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac30 0x7ff3500231f0 :-1 s=OPENED pgs=2175 cs=1 l=1).handle_message read tag failed 2021-03-09T10:39:31.878 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:31.872+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac30 0x7ff3500231f0 :-1 s=OPENED pgs=2175 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:39:31.879 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:31.872+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:39:33.414 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:33.412+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:39:35.212 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:35.209+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:36.832 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:36.832+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:36.835 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:36.832+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:36.877 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:36.873+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:39.821 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:39.820+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:39:40.214 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:40.213+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:41.832 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:41.832+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:41.835 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:41.836+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:41.877 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:41.876+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:45.236 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:45.232+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:46.833 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:46.832+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:46.836 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:46.836+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:46.877 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:46.876+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:50.238 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:50.236+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:51.833 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:51.832+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:51.836 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:51.836+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:51.877 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:51.876+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:52.634 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:52.632+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:39:55.260 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:39:55.256+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:39:56.833 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:39:56.831+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:39:56.836 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:39:56.835+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:39:56.877 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:39:56.872+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:39:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:39:57.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:39:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:39:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac30 legacy=0x7ff3500231f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350024910 legacy=0x7ff350026d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,1,4,5,8,7,3,6,0] 2021-03-09T10:39:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:39:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001cba0 addr v1:172.21.15.173:6789/0 2021-03-09T10:39:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35000ce20 addr v1:172.21.15.143:6791/0 2021-03-09T10:39:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35001cba0 2021-03-09T10:39:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:39:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35000ce20 2021-03-09T10:39:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:39:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:39:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.195+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:39:57.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.399+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:39:57.400 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.399+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:39:57.401 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.399+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:39:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.803+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:39:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.803+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:39:57.804 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:57.803+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:39:58.608 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:58.607+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:39:58.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:58.607+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:39:58.609 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:39:58.607+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:40:00.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:00.211+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:40:00.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:00.211+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:40:00.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:00.211+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:40:00.262 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:00.260+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:01.833 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:01.831+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:01.836 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:01.835+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:01.878 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:01.876+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:03.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:03.415+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:40:03.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:03.415+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:40:03.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:03.415+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:40:05.284 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:05.280+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:06.833 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:06.831+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:06.836 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:06.835+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:06.878 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:06.872+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:09.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:09.819+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:40:09.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:09.819+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:40:09.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:09.819+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:40:10.286 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:10.284+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:11.834 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:11.831+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:11.837 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:11.835+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:11.878 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:11.876+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:15.308 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:15.304+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:16.834 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:16.831+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:16.839 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:16.835+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:16.878 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:16.876+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:20.310 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:20.308+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:21.834 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:21.831+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:21.837 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:21.835+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:21.878 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:21.876+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:22.631 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:22.627+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:40:22.635 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:22.631+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:40:22.636 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:22.635+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:40:25.332 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:25.331+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:26.834 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:26.831+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:26.837 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:26.835+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:26.878 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:26.875+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:40:27.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:40:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:40:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,8,6,3,5,7] 2021-03-09T10:40:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35001cf80 addr v1:172.21.15.143:6789/0 2021-03-09T10:40:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:40:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350005aa0 addr v1:172.21.15.143:6793/0 2021-03-09T10:40:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001a820 addr v1:172.21.15.143:6792/0 2021-03-09T10:40:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cf80 2021-03-09T10:40:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35001a820 2021-03-09T10:40:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:40:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350005aa0 2021-03-09T10:40:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.199+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:40:27.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.403+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:40:27.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:27.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:40:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:28.611+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:40:30.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:30.215+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:40:30.334 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:30.331+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:31.834 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:31.834+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:31.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a820 legacy=0x7ff3500148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:31.836 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a820 legacy=0x7ff3500148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:31.836 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a820 0x7ff3500148f0 :-1 s=OPENED pgs=2258 cs=1 l=1).handle_message read tag failed 2021-03-09T10:40:31.836 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a820 0x7ff3500148f0 :-1 s=OPENED pgs=2258 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:40:31.836 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:40:31.837 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:31.834+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:31.838 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:31.838 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:31.839 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 0x7ff35001a110 :-1 s=OPENED pgs=2211 cs=1 l=1).handle_message read tag failed 2021-03-09T10:40:31.839 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 0x7ff35001a110 :-1 s=OPENED pgs=2211 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:40:31.839 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:31.834+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:40:31.879 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:31.879+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:33.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:33.418+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:40:35.356 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:35.355+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:36.836 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:36.830+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:36.838 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:36.834+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:36.879 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:36.879+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:39.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:39.822+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:40:40.376 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:40.375+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:41.835 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:41.830+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:41.838 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:41.834+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:41.879 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:41.879+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:45.378 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:45.375+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:46.835 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:46.830+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:46.838 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:46.834+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:46.879 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:46.879+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:50.400 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:50.399+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:51.835 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:51.834+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:51.838 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:51.834+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:51.880 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:51.879+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:52.639 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:52.634+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:40:55.402 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:40:55.399+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:40:56.835 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:40:56.830+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:40:56.838 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:40:56.834+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:40:56.880 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:40:56.879+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:40:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:40:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:40:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:40:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001a820 legacy=0x7ff3500148f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,3,7] 2021-03-09T10:40:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500163b0 addr v1:172.21.15.173:6791/0 2021-03-09T10:40:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:40:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001cba0 addr v1:172.21.15.173:6792/0 2021-03-09T10:40:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:40:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500163b0 2021-03-09T10:40:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35001cba0 2021-03-09T10:40:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.198+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:40:57.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.402+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:40:57.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:57.806+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:40:58.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:40:58.610+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:00.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:00.214+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:00.424 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:00.423+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:00.425 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:00.422+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:00.425 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:00.422+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:00.425 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:00.422+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=2221 cs=1 l=1).handle_message read tag failed 2021-03-09T10:41:00.425 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:00.422+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350006a70 :-1 s=OPENED pgs=2221 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:41:00.426 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:00.422+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:41:01.836 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:01.834+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:01.838 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:01.834+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:01.880 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:01.879+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:01.881 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:01.878+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:41:01.881 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:01.878+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:01.881 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:01.878+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 0x7ff350003380 :-1 s=OPENED pgs=2218 cs=1 l=1).handle_message read tag failed 2021-03-09T10:41:01.882 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:01.878+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 0x7ff350003380 :-1 s=OPENED pgs=2218 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:41:01.882 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:01.878+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:41:01.882 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:01.878+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:41:03.418 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:03.418+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:05.426 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:05.423+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:06.836 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:06.834+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:06.839 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:06.834+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:06.880 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:06.879+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:09.825 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:09.821+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:10.448 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:10.446+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:11.836 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:11.833+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:11.839 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:11.833+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:11.880 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:11.878+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:15.450 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:15.446+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:16.836 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:16.833+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:16.839 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:16.833+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:16.880 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:16.878+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:20.472 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:20.470+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:21.836 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:21.833+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:21.839 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:21.833+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:21.881 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:21.878+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:22.638 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:22.637+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:25.474 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:25.470+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:26.837 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:26.833+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:26.839 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:26.833+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:26.881 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:26.878+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:41:27.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:41:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:41:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cba0 legacy=0x7ff350003380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,2,8,6,4,5,0,7,1] 2021-03-09T10:41:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:41:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:41:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001cba0 addr v1:172.21.15.143:6793/0 2021-03-09T10:41:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:41:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:41:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cba0 2021-03-09T10:41:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:41:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.197+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:27.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.401+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:41:27.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.401+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:27.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.805+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:41:27.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:27.805+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:28.609+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:41:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:28.609+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:30.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:30.213+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:41:30.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:30.213+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:30.496 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:30.494+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:31.837 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:31.833+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:31.840 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:31.837+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:31.840 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:31.837+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:41:31.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:31.837+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:31.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:31.837+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 0x7ff350014160 :-1 s=OPENED pgs=2252 cs=1 l=1).handle_message read tag failed 2021-03-09T10:41:31.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:31.837+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 0x7ff350014160 :-1 s=OPENED pgs=2252 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:41:31.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:31.837+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:41:31.881 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:31.878+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:33.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:33.417+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:41:33.418 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:33.417+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:35.498 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:35.494+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:36.837 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:36.837+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:36.840 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:36.837+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:36.881 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:36.878+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:39.824 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:39.821+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:41:39.824 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:39.821+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:40.520 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:40.518+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:41.837 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:41.837+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:42.020 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:41.878+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:42.020 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:41.837+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:45.522 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:45.518+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:46.837 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:46.832+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:46.840 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:46.836+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:46.882 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:46.878+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:50.544 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:50.542+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:51.837 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:51.836+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:51.840 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:51.840+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:51.882 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:51.878+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:52.637 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:52.636+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:41:52.637 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:52.636+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:41:55.546 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:41:55.541+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:41:56.838 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:41:56.836+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:41:56.840 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:41:56.836+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:41:56.882 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:41:56.877+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:41:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:41:57.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:41:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:41:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 legacy=0x7ff350014160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,5,7,0,6,4] 2021-03-09T10:41:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35000ce20 addr v1:172.21.15.173:6789/0 2021-03-09T10:41:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001cf80 addr v1:172.21.15.173:6791/0 2021-03-09T10:41:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350005aa0 addr v1:172.21.15.173:6792/0 2021-03-09T10:41:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff35000ce20 2021-03-09T10:41:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cf80 2021-03-09T10:41:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350005aa0 2021-03-09T10:41:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.196+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:41:57.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.400+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:41:57.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:57.804+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:41:58.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:41:58.608+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:42:00.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:00.212+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:42:00.568 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:00.565+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:00.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:00.564+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:00.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:00.564+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:00.569 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:00.564+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cf80 0x7ff350005390 :-1 s=OPENED pgs=2262 cs=1 l=1).handle_message read tag failed 2021-03-09T10:42:00.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:00.564+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cf80 0x7ff350005390 :-1 s=OPENED pgs=2262 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:42:00.570 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:00.564+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:42:01.838 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:01.836+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:01.841 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:01.836+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:01.883 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:01.877+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:01.884 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:01.876+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:42:01.884 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:01.876+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:01.884 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:01.876+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 0x7ff35001a110 :-1 s=OPENED pgs=2254 cs=1 l=1).handle_message read tag failed 2021-03-09T10:42:01.885 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:01.876+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 0x7ff35001a110 :-1 s=OPENED pgs=2254 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:42:01.885 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:01.876+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:42:01.885 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:01.876+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:42:03.421 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:03.420+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:42:05.570 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:05.565+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:06.838 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:06.832+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:06.841 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:06.836+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:06.883 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:06.881+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:09.824+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:42:10.592 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:10.589+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:11.838 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:11.836+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:11.841 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:11.840+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:11.883 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:11.877+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:15.594 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:15.589+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:16.838 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:16.836+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:16.841 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:16.840+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:16.883 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:16.881+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:20.616 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:20.613+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:21.839 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:21.836+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:21.841 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:21.840+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:21.883 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:21.881+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:22.641 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:22.640+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:42:25.618 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:25.613+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:26.839 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:26.835+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:26.841 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:26.839+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:26.883 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:26.881+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:42:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:42:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:42:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,4,0] 2021-03-09T10:42:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001cf80 addr v1:172.21.15.143:6792/0 2021-03-09T10:42:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:42:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:42:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500163b0 2021-03-09T10:42:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff350005aa0 2021-03-09T10:42:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cf80 2021-03-09T10:42:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:42:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.199+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:27.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.403+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:27.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.403+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:42:27.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:27.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:27.807+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:42:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:28.611+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:42:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:28.611+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:30.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:30.215+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:30.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:30.215+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:42:30.640 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:30.637+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:31.839 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:31.835+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:31.840 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:31.835+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:31.840 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:31.835+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:31.840 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:31.835+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 0x7ff350005390 :-1 s=OPENED pgs=2318 cs=1 l=1).handle_message read tag failed 2021-03-09T10:42:31.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:31.835+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 0x7ff350005390 :-1 s=OPENED pgs=2318 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:42:31.841 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:31.835+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:42:31.842 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:31.839+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:31.883 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:31.881+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:33.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:33.419+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:42:33.421 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:33.419+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:35.642 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:35.637+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:36.839 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:36.839+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:36.842 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:36.839+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:36.884 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:36.881+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:39.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:39.823+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:42:39.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:39.823+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:40.664 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:40.660+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:41.839 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:41.839+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:41.842 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:41.839+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:41.884 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:41.880+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:45.666 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:45.664+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:46.839 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:46.835+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:46.842 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:46.839+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:46.884 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:46.880+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:50.688 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:50.684+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:51.840 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:51.839+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:51.842 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:51.839+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:51.884 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:51.884+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:52.637 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:52.635+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:42:52.640 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:52.639+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:55.690 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:42:55.688+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:42:56.840 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:42:56.839+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:42:56.842 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:42:56.839+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:42:56.884 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:42:56.884+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:42:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:42:57.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:42:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:42:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,0,1,3,2,4,8,7,6] 2021-03-09T10:42:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500111f0 addr v1:172.21.15.173:6791/0 2021-03-09T10:42:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:42:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35000ce20 addr v1:172.21.15.173:6789/0 2021-03-09T10:42:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:42:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35000ce20 2021-03-09T10:42:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc00 con 0x7ff3500111f0 2021-03-09T10:42:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.199+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:42:57.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.403+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:57.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.403+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:42:57.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.807+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:57.809 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:57.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:42:58.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:58.611+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:42:58.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:42:58.611+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:43:00.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:00.215+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:43:00.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:00.215+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:43:00.712 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:00.708+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:00.713 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:00.711+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:00.713 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:00.711+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:00.713 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:00.711+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=2293 cs=1 l=1).handle_message read tag failed 2021-03-09T10:43:00.714 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:00.711+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 0x7ff35001c490 :-1 s=OPENED pgs=2293 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:43:00.714 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:00.711+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:43:01.840 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:01.838+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:01.843 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:01.838+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:01.885 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:01.880+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:03.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:03.418+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:43:03.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:03.418+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:43:05.714 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:05.708+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:06.840 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:06.838+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:06.843 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:06.838+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:06.885 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:06.880+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:09.822 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:09.822+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:43:09.823 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:09.822+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:43:10.736 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:10.732+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:11.840 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:11.838+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:11.843 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:11.838+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:11.885 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:11.884+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:15.739 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:15.736+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:16.841 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:16.838+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:16.844 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:16.842+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:16.885 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:16.884+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:20.760 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:20.760+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:21.841 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:21.838+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:21.844 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:21.842+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:21.886 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:21.884+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:22.635 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:22.634+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:43:22.637 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:22.634+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:43:25.762 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:25.759+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:26.841 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:26.838+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:26.844 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:26.842+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:26.885 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:26.883+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:27.201 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:43:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:43:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:43:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35000ce20 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,6,2,4,7,3] 2021-03-09T10:43:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001cba0 addr v1:172.21.15.143:6793/0 2021-03-09T10:43:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:43:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001cf80 addr v1:172.21.15.143:6792/0 2021-03-09T10:43:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:43:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:43:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001cf80 2021-03-09T10:43:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cba0 2021-03-09T10:43:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.198+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:43:27.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.402+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:43:27.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:27.806+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:43:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:28.610+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:43:30.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:30.214+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:43:30.784 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:30.783+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:31.841 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:31.838+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:31.842 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.838+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:43:31.842 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.838+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:31.843 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.838+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 0x7ff350005390 :-1 s=OPENED pgs=2364 cs=1 l=1).handle_message read tag failed 2021-03-09T10:43:31.843 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.838+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 0x7ff350005390 :-1 s=OPENED pgs=2364 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:43:31.843 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.838+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:43:31.844 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:31.842+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:31.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.842+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:31.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.842+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:31.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.842+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 0x7ff35001a110 :-1 s=OPENED pgs=2310 cs=1 l=1).handle_message read tag failed 2021-03-09T10:43:31.845 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.842+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 0x7ff35001a110 :-1 s=OPENED pgs=2310 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:43:31.846 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:31.842+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:43:31.886 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:31.883+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:33.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:33.418+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:43:35.786 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:35.783+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:36.842 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:36.838+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:36.844 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:36.842+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:36.886 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:36.883+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:39.826 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:39.821+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:43:40.789 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:40.787+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:41.842 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:41.841+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:41.844 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:41.841+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:41.886 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:41.883+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:45.811 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:45.811+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:46.842 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:46.837+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:46.844 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:46.841+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:46.886 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:46.883+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:50.814 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:50.811+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:51.842 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:51.841+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:51.845 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:51.841+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:51.887 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:51.883+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:52.639 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:52.633+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:43:55.836 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:43:55.835+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:43:56.842 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:43:56.837+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:43:56.845 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:43:56.841+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:43:56.887 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:43:56.887+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:43:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:43:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:43:57.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:43:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,7,4] 2021-03-09T10:43:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350005aa0 addr v1:172.21.15.173:6790/0 2021-03-09T10:43:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:43:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35000ce20 addr v1:172.21.15.143:6791/0 2021-03-09T10:43:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff350005aa0 2021-03-09T10:43:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35000ce20 2021-03-09T10:43:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500111f0 2021-03-09T10:43:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:43:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.201+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:43:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.405+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:43:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.405+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:43:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:43:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:57.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:43:58.613 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:58.613+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:43:58.614 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:43:58.613+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:00.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:00.217+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:44:00.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:00.217+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:00.838 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:00.835+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:01.843 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:01.841+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:01.845 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:01.841+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:01.887 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:01.887+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:01.888 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:01.881+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:01.888 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:01.881+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:01.888 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:01.881+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350014160 :-1 s=OPENED pgs=2327 cs=1 l=1).handle_message read tag failed 2021-03-09T10:44:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:01.881+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350014160 :-1 s=OPENED pgs=2327 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:44:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:01.881+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:44:03.423 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:03.421+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:44:03.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:03.421+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:05.860 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:05.859+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:06.843 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:06.841+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:06.845 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:06.841+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:06.887 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:06.887+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:09.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:09.829+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:44:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:09.829+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:10.862 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:10.858+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:11.843 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:11.841+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:11.845 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:11.845+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:11.888 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:11.886+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:15.884 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:15.882+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:16.843 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:16.841+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:16.846 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:16.845+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:16.888 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:16.886+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:20.886 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:20.882+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:21.844 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:21.840+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:21.846 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:21.844+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:21.888 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:21.886+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:22.640+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:44:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:22.640+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:25.908 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:25.906+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:26.844 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:26.836+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:26.846 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:26.840+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:26.888 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:26.886+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:27.202 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:44:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:44:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:44:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350014160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35000ce20 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350005aa0 legacy=0x7ff3500148d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,3,0,1,4,6,5,2,7] 2021-03-09T10:44:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:44:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff35001cba0 addr v1:172.21.15.143:6793/0 2021-03-09T10:44:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:44:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:44:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:44:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500111f0 2021-03-09T10:44:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cba0 2021-03-09T10:44:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.196+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:44:27.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.400+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:44:27.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.400+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:27.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.804+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:44:27.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:27.804+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:28.608+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:44:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:28.608+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:30.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:30.212+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:44:30.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:30.212+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:30.910 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:30.906+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:31.844 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:31.840+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:31.846 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:31.840+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:31.847 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:31.840+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:31.847 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:31.840+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:31.848 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:31.840+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 0x7ff35001a110 :-1 s=OPENED pgs=2356 cs=1 l=1).handle_message read tag failed 2021-03-09T10:44:31.848 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:31.840+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 0x7ff35001a110 :-1 s=OPENED pgs=2356 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:44:31.848 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:31.840+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:44:31.888 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:31.886+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:33.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:33.416+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:33.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:33.416+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:44:35.932 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:35.930+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:36.844 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:36.844+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:36.847 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:36.844+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:36.889 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:36.886+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:39.824 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:39.824+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:39.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:39.824+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:44:40.935 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:40.930+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:41.844 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:41.844+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:41.847 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:41.844+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:41.889 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:41.886+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:45.956 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:45.954+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:46.844 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:46.844+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:46.847 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:46.844+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:46.889 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:46.886+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:50.958 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:50.954+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:51.844 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:51.844+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:51.847 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:51.844+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:51.889 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:51.886+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:52.637 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:52.636+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:44:52.640 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:52.640+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:44:55.980 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:44:55.978+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:44:56.845 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:44:56.843+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:44:56.847 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:44:56.843+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:44:56.889 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:44:56.890+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:44:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:44:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:44:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:44:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff35001cba0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,5,4,7,2,6] 2021-03-09T10:44:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001cf80 addr v1:172.21.15.173:6789/0 2021-03-09T10:44:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35000ce10 addr v1:172.21.15.173:6791/0 2021-03-09T10:44:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350015130 addr v1:172.21.15.143:6791/0 2021-03-09T10:44:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cf80 2021-03-09T10:44:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350015130 2021-03-09T10:44:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff35000ce10 2021-03-09T10:44:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:44:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.199+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:44:57.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.403+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:44:57.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.403+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:44:57.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:44:57.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:57.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:44:58.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:58.611+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:44:58.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:44:58.611+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:45:00.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:00.215+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:45:00.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:00.215+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:45:00.983 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:00.977+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:00.983 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:00.979+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:45:00.984 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:00.979+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:00.984 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:00.979+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 0x7ff350014160 :-1 s=OPENED pgs=2372 cs=1 l=1).handle_message read tag failed 2021-03-09T10:45:00.984 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:00.979+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 0x7ff350014160 :-1 s=OPENED pgs=2372 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:45:00.984 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:00.979+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:45:01.845 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:01.843+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:01.847 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:01.847+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:01.890 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:01.885+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:03.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:03.419+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:45:03.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:03.419+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:45:06.004 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:06.001+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:06.845 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:06.843+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:06.848 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:06.847+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:06.890 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:06.885+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:09.823+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:45:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:09.823+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:45:11.006 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:11.001+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:11.846 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:11.843+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:11.848 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:11.847+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:11.890 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:11.889+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:16.028 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:16.025+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:16.846 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:16.843+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:16.848 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:16.847+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:16.890 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:16.889+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:21.030 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:21.025+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:21.846 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:21.843+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:21.848 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:21.847+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:21.890 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:21.885+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:22.640 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:22.639+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:45:22.640 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:22.639+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:45:26.052 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:26.049+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:26.846 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:26.843+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:26.848 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:26.847+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:26.891 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:26.885+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:45:27.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:45:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:45:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35000ce10 legacy=0x7ff350014160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff350014870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,6,7] 2021-03-09T10:45:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500053a0 addr v1:172.21.15.143:6790/0 2021-03-09T10:45:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:45:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350024b00 addr v1:172.21.15.143:6792/0 2021-03-09T10:45:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:45:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500053a0 2021-03-09T10:45:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350024b00 2021-03-09T10:45:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:45:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.199+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff350005780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:45:27.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.403+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff350005780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:45:27.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:27.807+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff350005780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:45:28.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:28.611+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff350005780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:45:30.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:30.215+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff350005780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:45:31.054 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:31.049+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:31.846 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:31.843+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:31.847 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.843+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 legacy=0x7ff350026ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:45:31.847 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.843+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 legacy=0x7ff350026ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:31.848 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.843+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 0x7ff350026ef0 :-1 s=OPENED pgs=2436 cs=1 l=1).handle_message read tag failed 2021-03-09T10:45:31.848 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.843+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 0x7ff350026ef0 :-1 s=OPENED pgs=2436 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:45:31.848 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.843+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:45:31.849 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:31.847+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:31.891 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:31.889+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:31.892 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.887+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:31.892 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.887+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:31.892 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.887+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=2389 cs=1 l=1).handle_message read tag failed 2021-03-09T10:45:31.892 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.887+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=2389 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:45:31.893 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:31.887+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:45:33.421 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:33.419+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff350005780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:45:36.076 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:36.073+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:36.847 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:36.842+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:36.849 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:36.846+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:36.891 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:36.889+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:39.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:39.822+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff350005780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:45:41.078 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:41.077+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:41.847 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:41.842+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:41.849 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:41.846+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:41.891 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:41.889+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:46.100 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:46.096+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:46.847 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:46.842+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:46.849 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:46.846+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:46.891 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:46.888+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:51.103 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:51.096+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:51.847 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:51.846+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:51.849 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:51.846+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:51.891 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:51.888+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:52.640 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:52.638+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff350005780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:45:56.124 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:45:56.120+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:45:56.847 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:45:56.842+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:45:56.849 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:45:56.846+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:45:56.892 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:45:56.892+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:45:57.203 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:45:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:45:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:45:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 legacy=0x7ff350026ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500053a0 legacy=0x7ff350005780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,7,3,8,0,1,6,2,5] 2021-03-09T10:45:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350015130 addr v1:172.21.15.143:6791/0 2021-03-09T10:45:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500111f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:45:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500163b0 addr v1:172.21.15.173:6790/0 2021-03-09T10:45:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500163b0 2021-03-09T10:45:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350015130 2021-03-09T10:45:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500111f0 2021-03-09T10:45:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:45:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.198+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:45:57.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.402+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:45:57.404 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.402+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:45:57.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.806+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:45:57.808 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:57.806+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:45:58.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:58.610+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:45:58.612 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:45:58.610+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:46:00.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:00.214+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:46:00.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:00.214+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:46:01.126 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:01.124+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:01.848 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:01.842+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:01.850 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:01.846+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:01.892 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:01.888+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:01.893 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:01.890+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:01.893 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:01.890+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:01.893 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:01.890+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35001cba0 :-1 s=OPENED pgs=2406 cs=1 l=1).handle_message read tag failed 2021-03-09T10:46:01.893 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:01.890+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 0x7ff35001cba0 :-1 s=OPENED pgs=2406 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:46:01.894 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:01.890+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:46:03.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:03.418+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:46:03.420 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:03.418+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:46:06.148 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:06.144+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:06.848 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:06.846+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:06.850 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:06.846+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:06.892 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:06.892+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:09.826+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:46:09.827 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:09.826+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:46:11.150 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:11.148+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:11.848 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:11.846+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:11.850 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:11.846+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:11.892 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:11.892+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:16.172 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:16.168+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:16.848 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:16.845+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:16.850 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:16.845+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:16.892 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:16.892+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:21.174 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:21.168+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:21.848 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:21.845+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:21.850 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:21.845+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:21.893 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:21.892+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:22.640 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:22.637+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:46:22.641 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:22.637+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:46:26.196 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:26.196+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:26.848 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:26.845+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:26.851 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:26.845+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:26.893 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:26.892+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:46:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:46:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:46:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015130 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,5,6,0,1,2] 2021-03-09T10:46:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:46:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350012150 addr v1:172.21.15.143:6793/0 2021-03-09T10:46:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350018100 addr v1:172.21.15.173:6791/0 2021-03-09T10:46:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350024b00 addr v1:172.21.15.143:6792/0 2021-03-09T10:46:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350018100 2021-03-09T10:46:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff350024b00 2021-03-09T10:46:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350012150 2021-03-09T10:46:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:27.197+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:31.199 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:31.195+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:31.199 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.197+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350018100 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:31.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.197+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350018100 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:31.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.197+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350018100 0x7ff35001a520 :-1 s=OPENED pgs=2427 cs=1 l=1).handle_message read tag failed 2021-03-09T10:46:31.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.197+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350018100 0x7ff35001a520 :-1 s=OPENED pgs=2427 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:46:31.200 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.197+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:46:31.849 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:31.845+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:31.851 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:46:31.851 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:31.852 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 0x7ff350020ad0 :-1 s=OPENED pgs=2478 cs=1 l=1).handle_message read tag failed 2021-03-09T10:46:31.852 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 0x7ff350020ad0 :-1 s=OPENED pgs=2478 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:46:31.852 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:46:31.854 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:31.845+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:31.854 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:31.855 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:31.855 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350012150 0x7ff350014540 :-1 s=OPENED pgs=2428 cs=1 l=1).handle_message read tag failed 2021-03-09T10:46:31.856 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350012150 0x7ff350014540 :-1 s=OPENED pgs=2428 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:46:31.856 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:31.845+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:46:31.893 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:31.891+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:36.201 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:36.199+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:36.849 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:36.845+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:36.851 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:36.845+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:36.893 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:36.891+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:41.220 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:41.219+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:41.849 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:41.849+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:41.851 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:41.849+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:41.896 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:41.891+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:46.222 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:46.219+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:46.849 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:46.845+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:46.851 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:46.849+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:46.894 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:46.891+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:51.244 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:51.243+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:51.850 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:51.844+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:51.852 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:51.848+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:51.894 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:51.891+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:56.246 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:46:56.243+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:46:56.850 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:46:56.848+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:46:56.852 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:46:56.848+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:46:56.894 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:46:56.891+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:46:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:46:57.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:46:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:46:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350012150 legacy=0x7ff350014540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350024b00 legacy=0x7ff350020ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350018100 legacy=0x7ff35001a520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:46:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500163b0 addr v1:172.21.15.173:6789/0 2021-03-09T10:46:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T10:46:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350015130 addr v1:172.21.15.143:6789/0 2021-03-09T10:46:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350015130 2021-03-09T10:46:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500163b0 2021-03-09T10:46:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:46:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015130 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:46:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:46:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.200+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:46:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.404+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:46:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.404+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015130 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:46:57.409 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.404+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:46:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.808+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:46:57.871 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.808+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015130 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:46:57.871 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:57.808+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:46:58.613 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:58.612+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:46:58.613 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:58.612+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015130 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:46:58.614 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:46:58.612+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:47:00.222 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:00.216+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:00.222 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:00.216+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015130 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:47:00.223 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:00.216+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:47:01.268 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:01.267+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:01.850 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:01.848+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:01.852 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:01.848+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:01.894 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:01.891+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:03.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:03.420+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015130 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:47:03.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:03.420+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:03.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:03.420+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:47:06.270 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:06.267+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:06.850 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:06.844+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:06.852 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:06.848+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:06.894 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:06.891+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:09.828+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:09.828+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015130 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:47:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:09.828+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:47:11.292 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:11.291+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:11.850 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:11.848+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:11.853 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:11.852+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:11.895 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:11.891+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:16.294 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:16.291+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:16.851 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:16.848+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:16.853 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:16.852+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:16.895 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:16.895+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:21.316 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:21.314+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:21.851 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:21.848+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:21.853 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:21.852+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:21.895 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:21.894+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:22.640+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:22.644+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015130 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:47:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:22.644+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:47:26.319 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:26.314+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:26.851 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:26.848+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:26.853 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:26.852+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:26.895 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:26.894+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:27.204 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:47:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:47:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:47:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350015130 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:47:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,2,3,0,5,4,8,7,1] 2021-03-09T10:47:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff3500111f0 addr v1:172.21.15.143:6792/0 2021-03-09T10:47:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:47:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500150a0 addr v1:172.21.15.173:6790/0 2021-03-09T10:47:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:47:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500150a0 2021-03-09T10:47:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500111f0 2021-03-09T10:47:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.204+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500150a0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:47:27.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.408+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:27.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.408+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500150a0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:47:27.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.812+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:27.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:27.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500150a0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:47:28.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:28.616+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:28.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:28.616+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500150a0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:47:30.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:30.219+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:30.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:30.219+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500150a0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:47:31.340 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:31.338+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:31.851 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:31.847+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:31.852 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:31.847+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:31.853 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:31.847+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:31.853 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:31.847+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=2517 cs=1 l=1).handle_message read tag failed 2021-03-09T10:47:31.853 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:31.847+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=2517 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:47:31.853 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:31.847+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:47:31.854 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:31.851+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:31.895 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:31.894+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:33.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:33.423+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:33.425 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:33.423+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500150a0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:47:36.343 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:36.338+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:36.851 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:36.851+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:36.853 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:36.851+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:36.896 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:36.894+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:39.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:39.827+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:39.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:39.831+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500150a0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:47:41.364 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:41.362+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:41.852 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:41.851+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:41.854 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:41.851+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:41.896 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:41.894+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:46.367 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:46.362+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:46.852 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:46.851+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:46.854 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:46.851+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:46.896 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:46.894+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:51.388 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:51.386+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:51.852 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:51.851+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:51.854 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:51.851+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:51.896 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:51.894+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:52.643 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:52.639+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:47:52.645 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:52.643+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500150a0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:47:56.391 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:47:56.386+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:47:56.852 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:47:56.851+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:47:56.854 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:47:56.851+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:47:56.897 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:47:56.894+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:47:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:47:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:47:57.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:47:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500150a0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[0,4,7,1,8,5] 2021-03-09T10:47:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350005390 addr v1:172.21.15.143:6789/0 2021-03-09T10:47:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350014160 addr v1:172.21.15.143:6791/0 2021-03-09T10:47:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350014c80 addr v1:172.21.15.173:6792/0 2021-03-09T10:47:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350005390 2021-03-09T10:47:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350014160 2021-03-09T10:47:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350014c80 2021-03-09T10:47:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014160 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:47:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.203+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005390 legacy=0x7ff350005770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:47:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.407+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014160 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:47:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.407+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005390 legacy=0x7ff350005770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:47:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.811+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014160 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:47:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:57.811+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005390 legacy=0x7ff350005770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:47:58.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:58.615+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014160 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:47:58.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:47:58.615+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005390 legacy=0x7ff350005770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:48:00.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:00.219+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014160 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:48:00.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:00.219+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005390 legacy=0x7ff350005770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:48:01.412 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:01.409+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:01.852 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:01.851+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:01.854 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:01.851+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:01.897 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:01.893+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:01.898 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:01.895+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350014c80 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:01.898 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:01.895+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350014c80 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:01.898 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:01.895+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350014c80 0x7ff35001a110 :-1 s=OPENED pgs=2484 cs=1 l=1).handle_message read tag failed 2021-03-09T10:48:01.899 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:01.895+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350014c80 0x7ff35001a110 :-1 s=OPENED pgs=2484 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:48:01.899 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:01.895+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:48:03.423 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:03.423+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005390 legacy=0x7ff350005770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:48:03.423 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:03.423+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014160 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:48:06.423 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:06.409+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:06.853 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:06.850+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:06.855 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:06.850+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:06.897 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:06.893+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:09.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:09.826+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005390 legacy=0x7ff350005770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:48:09.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:09.826+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014160 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:48:11.436 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:11.433+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:11.853 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:11.850+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:11.855 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:11.850+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:11.897 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:11.897+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:16.438 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:16.433+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:16.853 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:16.850+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:16.855 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:16.850+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:16.897 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:16.893+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:21.460 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:21.457+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:21.853 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:21.850+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:21.855 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:21.850+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:21.898 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:21.897+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:22.642+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005390 legacy=0x7ff350005770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:48:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:22.642+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014160 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:48:26.463 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:26.457+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:26.853 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:26.850+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:26.855 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:26.850+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:26.898 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:26.893+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:27.205 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:48:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:48:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:48:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350014c80 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:27.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014160 legacy=0x7ff350014570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350005390 legacy=0x7ff350005770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,1,8] 2021-03-09T10:48:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:48:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001a820 addr v1:172.21.15.173:6791/0 2021-03-09T10:48:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:48:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500111f0 addr v1:172.21.15.173:6789/0 2021-03-09T10:48:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500163b0 addr v1:172.21.15.143:6793/0 2021-03-09T10:48:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500111f0 2021-03-09T10:48:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001a820 2021-03-09T10:48:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:48:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.202+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:27.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.406+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:27.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:27.810+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:28.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:28.614+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:30.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:30.218+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:31.484 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:31.481+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:31.485 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.482+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:31.486 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.482+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:31.486 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.482+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 0x7ff350023af0 :-1 s=OPENED pgs=2501 cs=1 l=1).handle_message read tag failed 2021-03-09T10:48:31.486 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.482+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 0x7ff350023af0 :-1 s=OPENED pgs=2501 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:48:31.486 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.482+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:48:31.854 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:31.850+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:31.856 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:31.854+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:31.856 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.854+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:48:31.856 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.854+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:31.857 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.854+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35001c490 :-1 s=OPENED pgs=2491 cs=1 l=1).handle_message read tag failed 2021-03-09T10:48:31.857 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.854+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35001c490 :-1 s=OPENED pgs=2491 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:48:31.857 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:31.854+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:48:31.898 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:31.893+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:33.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:33.422+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:36.487 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:36.481+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:36.854 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:36.854+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:36.856 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:36.854+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:36.898 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:36.897+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:39.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:39.826+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:41.508 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:41.505+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:41.854 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:41.850+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:41.856 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:41.854+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:41.898 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:41.897+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:46.511 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:46.508+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:46.854 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:46.849+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:46.856 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:46.853+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:46.899 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:46.892+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:51.532 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:51.528+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:51.854 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:51.849+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:51.856 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:51.853+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:51.899 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:51.896+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:52.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:52.641+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:56.535 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:48:56.532+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:48:56.854 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:48:56.853+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:48:56.856 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:48:56.853+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:48:56.899 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:48:56.896+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:48:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:48:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:48:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:48:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,0,5,6,3,7,4,2,8] 2021-03-09T10:48:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001ac00 addr v1:172.21.15.173:6789/0 2021-03-09T10:48:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500111f0 addr v1:172.21.15.143:6789/0 2021-03-09T10:48:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001a820 addr v1:172.21.15.173:6791/0 2021-03-09T10:48:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015580 con 0x7ff3500111f0 2021-03-09T10:48:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff35001ac00 2021-03-09T10:48:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001a820 2021-03-09T10:48:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:48:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.201+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001ac00 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.405+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:48:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.405+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001ac00 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.809+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:48:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:57.809+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001ac00 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:48:58.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:58.613+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:48:58.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:48:58.613+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001ac00 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:49:00.218 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:00.217+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:49:00.218 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:00.217+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001ac00 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:49:01.557 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:01.553+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:49:01.557 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:01.553+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:01.557 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:01.553+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 0x7ff350005390 :-1 s=OPENED pgs=2519 cs=1 l=1).handle_message read tag failed 2021-03-09T10:49:01.558 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:01.553+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 0x7ff350005390 :-1 s=OPENED pgs=2519 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:49:01.558 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:01.553+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:49:01.559 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:01.552+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:01.855 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:01.853+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:01.857 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:01.853+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:01.899 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:01.892+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:03.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:03.421+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:49:03.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:03.421+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001ac00 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:49:06.559 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:06.556+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:06.855 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:06.853+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:06.857 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:06.853+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:06.899 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:06.896+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:09.829+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:49:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:09.829+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001ac00 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:49:11.585 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:11.580+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:11.855 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:11.849+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:11.857 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:11.857+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:11.900 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:11.900+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:16.587 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:16.584+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:16.855 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:16.853+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:16.857 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:16.853+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:16.900 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:16.900+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:21.608 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:21.608+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:21.855 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:21.848+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:21.857 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:21.856+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:21.900 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:21.900+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:22.636+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:49:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:22.640+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001ac00 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:49:26.611 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:26.607+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:26.856 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:26.848+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:26.857 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:26.852+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:26.900 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:26.899+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:49:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:49:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:49:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001ac00 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,8,6,3,2,4] 2021-03-09T10:49:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350005aa0 addr v1:172.21.15.173:6792/0 2021-03-09T10:49:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500163b0 addr v1:172.21.15.143:6793/0 2021-03-09T10:49:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350015180 addr v1:172.21.15.143:6792/0 2021-03-09T10:49:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350015180 2021-03-09T10:49:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015580 con 0x7ff350005aa0 2021-03-09T10:49:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500163b0 2021-03-09T10:49:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:27.204+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:31.613 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:31.611+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:31.856 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:31.852+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:31.857 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015180 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:49:31.857 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015180 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:31.857 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015180 0x7ff35001fac0 :-1 s=OPENED pgs=2575 cs=1 l=1).handle_message read tag failed 2021-03-09T10:49:31.857 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015180 0x7ff35001fac0 :-1 s=OPENED pgs=2575 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:49:31.858 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:49:31.858 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:31.852+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:31.859 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:31.859 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:31.859 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35001c490 :-1 s=OPENED pgs=2520 cs=1 l=1).handle_message read tag failed 2021-03-09T10:49:31.859 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 0x7ff35001c490 :-1 s=OPENED pgs=2520 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:49:31.860 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:49:31.860 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.852+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:49:31.900 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:31.899+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:31.901 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.896+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:31.901 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.896+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:31.901 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.896+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 0x7ff350019100 :-1 s=OPENED pgs=2526 cs=1 l=1).handle_message read tag failed 2021-03-09T10:49:31.902 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.896+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 0x7ff350019100 :-1 s=OPENED pgs=2526 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:49:31.902 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:31.896+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:49:36.632 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:36.631+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:36.856 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:36.856+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:36.858 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:36.856+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:36.901 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:36.899+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:41.635 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:41.631+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:41.856 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:41.856+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:41.858 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:41.856+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:41.901 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:41.899+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:46.660 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:46.659+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:46.857 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:46.856+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:46.858 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:46.856+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:46.901 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:46.899+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:51.663 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:51.659+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:51.857 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:51.856+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:51.859 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:51.856+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:51.901 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:51.899+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:56.684 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:49:56.683+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:49:56.857 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:49:56.855+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:49:56.859 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:49:56.855+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:49:56.901 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:49:56.899+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:49:57.206 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:49:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:49:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:49:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff350019100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015180 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,4,2] 2021-03-09T10:49:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff350015180 addr v1:172.21.15.173:6790/0 2021-03-09T10:49:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500163b0 addr v1:172.21.15.143:6791/0 2021-03-09T10:49:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T10:49:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:49:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015560 con 0x7ff350015180 2021-03-09T10:49:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500163b0 2021-03-09T10:49:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:49:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:49:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.203+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015180 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:49:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.407+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:49:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.407+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:49:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.407+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015180 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:49:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.811+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:49:57.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.811+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:49:57.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:57.811+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015180 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:49:58.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:58.615+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:49:58.617 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:58.615+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:49:58.617 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:49:58.615+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015180 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:50:00.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:00.219+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:50:00.221 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:00.219+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:00.221 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:00.219+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015180 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:50:01.687 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:01.683+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:01.857 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:01.855+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:01.859 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:01.855+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:01.901 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:01.899+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:03.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:03.423+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:50:03.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:03.423+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:03.425 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:03.423+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015180 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:50:06.708 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:06.707+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:06.857 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:06.855+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:06.859 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:06.855+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:06.902 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:06.899+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:09.827+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:50:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:09.827+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:09.827+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015180 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:50:11.711 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:11.706+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:11.857 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:11.855+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:11.859 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:11.855+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:11.902 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:11.898+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:16.732 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:16.730+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:16.858 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:16.855+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:16.860 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:16.855+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:16.902 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:16.898+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:21.735 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:21.730+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:21.858 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:21.855+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:21.860 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:21.859+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:21.903 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:21.902+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:22.643+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:50:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:22.643+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:22.643+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015180 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:50:26.756 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:26.754+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:26.858 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:26.855+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:26.860 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:26.859+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:26.903 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:26.898+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:50:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:50:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:50:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001a820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff350015180 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,5,1,7,8,3,4,6,0] 2021-03-09T10:50:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500111f0 addr v1:172.21.15.143:6790/0 2021-03-09T10:50:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff35001c490 addr v1:172.21.15.173:6791/0 2021-03-09T10:50:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001cf80 addr v1:172.21.15.173:6789/0 2021-03-09T10:50:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cf80 2021-03-09T10:50:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:50:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350015560 con 0x7ff35001c490 2021-03-09T10:50:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.203+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:50:27.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.407+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:27.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.407+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:50:27.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.811+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:27.813 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:27.811+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:50:28.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:28.615+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:28.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:28.615+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:50:30.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:30.219+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:30.221 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:30.219+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:50:31.759 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:31.754+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:31.760 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:31.755+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:31.760 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:31.755+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:31.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:31.755+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c490 0x7ff35001c870 :-1 s=OPENED pgs=2569 cs=1 l=1).handle_message read tag failed 2021-03-09T10:50:31.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:31.755+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c490 0x7ff35001c870 :-1 s=OPENED pgs=2569 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:50:31.761 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:31.755+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:50:31.859 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:31.855+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:31.860 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:31.859+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:31.903 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:31.898+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:32.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:32.291+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 legacy=0x7ff3540326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:50:32.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:32.291+0000 7ff372ffd700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 legacy=0x7ff3540326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:32.296 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:32.291+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 0x7ff3540326f0 :-1 s=OPENED pgs=85 cs=1 l=1).handle_message read tag failed 2021-03-09T10:50:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:32.291+0000 7ff372ffd700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 0x7ff3540326f0 :-1 s=OPENED pgs=85 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:50:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:32.291+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:50:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:32.291+0000 7ff35f7fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6812/16455 conn(0x7ff354030290 legacy=0x7ff3540326f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:32.297 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:32.295+0000 7ff372ffd700 10 monclient(hunting): get_auth_request con 0x7ff354043c70 auth_method 0 2021-03-09T10:50:33.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:33.423+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:33.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:33.423+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:50:36.780 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:36.778+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:36.859 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:36.858+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:36.860 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:36.858+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:36.903 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:36.898+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:39.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:39.826+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:39.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:39.826+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:50:41.783 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:41.778+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:41.859 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:41.858+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:41.860 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:41.858+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:41.903 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:41.902+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:46.804 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:46.802+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:46.859 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:46.858+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:46.860 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:46.858+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:46.903 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:46.902+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:51.807 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:51.802+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:51.859 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:51.854+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:51.861 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:51.858+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:51.904 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:51.902+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:52.640 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:52.638+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:50:52.641 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:52.638+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:50:56.829 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:50:56.825+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:50:56.859 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:50:56.854+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:50:56.861 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:50:56.858+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:50:56.904 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:50:56.901+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:50:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:50:57.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:50:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:50:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001cf80 legacy=0x7ff35001fac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,0,7,8,6,4] 2021-03-09T10:50:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff35001c490 addr v1:172.21.15.173:6790/0 2021-03-09T10:50:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:50:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff350005aa0 addr v1:172.21.15.173:6792/0 2021-03-09T10:50:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500163b0 2021-03-09T10:50:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001c490 2021-03-09T10:50:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350005aa0 2021-03-09T10:50:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:50:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.202+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:50:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.406+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:50:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.406+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:50:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.810+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:50:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:57.810+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:50:58.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:58.614+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:50:58.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:50:58.614+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:51:00.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:00.218+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:51:00.221 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:00.218+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:51:01.831 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:01.825+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:01.859 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:01.854+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:01.861 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:01.858+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:01.904 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:01.901+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:01.905 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:01.902+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:51:01.906 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:01.902+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:01.906 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:01.902+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 0x7ff350014160 :-1 s=OPENED pgs=2590 cs=1 l=1).handle_message read tag failed 2021-03-09T10:51:01.906 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:01.902+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 0x7ff350014160 :-1 s=OPENED pgs=2590 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:51:01.906 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:01.902+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:51:03.422 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:03.422+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:51:03.423 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:03.422+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:51:06.852 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:06.849+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:06.859 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:06.858+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:06.861 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:06.858+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:06.904 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:06.901+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:09.829 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:09.826+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:51:09.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:09.826+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:51:11.855 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:11.849+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:11.860 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:11.857+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:11.873 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:11.869+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:11.905 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:11.901+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:16.860 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:16.857+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:16.876 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:16.873+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:16.877 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:16.873+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:16.905 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:16.901+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:21.860 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:21.857+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:21.877 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:21.873+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:21.879 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:21.873+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:21.905 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:21.905+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:22.642 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:22.641+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:51:22.643 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:22.641+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:51:26.861 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:26.857+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:26.877 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:26.873+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:26.900 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:26.897+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:26.905 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:26.901+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:27.207 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:51:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:51:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:51:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff350005aa0 legacy=0x7ff350014160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff35001c490 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,4,8] 2021-03-09T10:51:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350014870 addr v1:172.21.15.143:6792/0 2021-03-09T10:51:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:51:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff35001a110 addr v1:172.21.15.143:6791/0 2021-03-09T10:51:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350025210 addr v1:172.21.15.143:6793/0 2021-03-09T10:51:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001a110 2021-03-09T10:51:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff350014870 2021-03-09T10:51:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350025210 2021-03-09T10:51:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.201+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:51:27.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.405+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:51:27.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:27.809+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:51:28.616 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:28.613+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:51:30.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:30.217+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:51:31.861 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:31.857+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:31.862 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.857+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350014870 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:31.862 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.857+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350014870 legacy=0x7ff350014c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:31.862 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.857+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350014870 0x7ff350014c50 :-1 s=OPENED pgs=2652 cs=1 l=1).handle_message read tag failed 2021-03-09T10:51:31.862 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.857+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350014870 0x7ff350014c50 :-1 s=OPENED pgs=2652 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:51:31.863 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.857+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:51:31.877 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:31.873+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:31.878 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.873+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 legacy=0x7ff350027600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:51:31.878 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.873+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 legacy=0x7ff350027600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:31.878 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.873+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 0x7ff350027600 :-1 s=OPENED pgs=2594 cs=1 l=1).handle_message read tag failed 2021-03-09T10:51:31.879 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.873+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 0x7ff350027600 :-1 s=OPENED pgs=2594 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:51:31.879 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:31.873+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:51:31.903 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:31.897+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:31.905 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:31.901+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:33.423 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:33.421+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:51:36.861 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:36.861+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:36.877 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:36.873+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:36.905 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:36.901+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:36.925 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:36.921+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:39.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:39.829+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:51:41.861 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:41.857+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:41.878 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:41.873+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:41.906 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:41.904+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:41.927 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:41.924+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:46.861 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:46.857+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:46.878 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:46.873+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:46.906 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:46.900+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:46.948 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:46.944+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:51.862 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:51.860+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:51.878 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:51.872+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:51.906 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:51.904+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:51.951 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:51.948+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:52.643 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:52.640+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:51:56.862 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:51:56.860+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:51:56.878 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:51:56.876+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:51:56.906 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:51:56.900+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:51:56.973 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:51:56.968+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:51:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:51:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:51:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:51:57.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 legacy=0x7ff350027600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350014870 legacy=0x7ff350014c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff35001a110 legacy=0x7ff35001a520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,3,8,5,2,7,6,0,4] 2021-03-09T10:51:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500111f0 addr v1:172.21.15.173:6789/0 2021-03-09T10:51:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500163b0 addr v1:172.21.15.173:6790/0 2021-03-09T10:51:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350025210 addr v1:172.21.15.143:6793/0 2021-03-09T10:51:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc90 con 0x7ff3500111f0 2021-03-09T10:51:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500163b0 2021-03-09T10:51:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff350025210 2021-03-09T10:51:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:51:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.200+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:51:57.408 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.404+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:51:57.409 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.404+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:51:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.808+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:51:57.812 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:57.808+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:51:58.613 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:58.612+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:51:58.614 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:51:58.612+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:52:00.220 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:00.216+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:52:00.221 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:00.216+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:52:01.862 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:01.860+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:01.878 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:01.872+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:01.879 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:01.872+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:52:01.879 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:01.872+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:01.879 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:01.872+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 0x7ff35001cba0 :-1 s=OPENED pgs=2613 cs=1 l=1).handle_message read tag failed 2021-03-09T10:52:01.880 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:01.872+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 0x7ff35001cba0 :-1 s=OPENED pgs=2613 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:52:01.880 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:01.872+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:52:01.906 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:01.900+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:01.975 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:01.972+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:03.424 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:03.420+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:52:03.425 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:03.420+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:52:06.862 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:06.856+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:06.879 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:06.872+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:06.906 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:06.900+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:06.997 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:06.992+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:09.828 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:09.824+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:52:09.831 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:09.824+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:52:11.862 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:11.860+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:11.879 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:11.872+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:11.907 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:11.900+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:11.999 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:11.992+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:16.863 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:16.860+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:16.879 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:16.876+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:16.907 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:16.904+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:17.021 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:17.020+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:21.863 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:21.860+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:21.879 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:21.876+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:21.907 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:21.904+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:22.023 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:22.020+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:22.641 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:22.640+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:52:22.644 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:22.644+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:52:26.863 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:26.859+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:26.879 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:26.875+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:26.907 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:26.903+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:27.045 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:27.043+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:27.208 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:52:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:52:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:52:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350025210 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,4,0,7,2,6] 2021-03-09T10:52:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:52:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350013100 addr v1:172.21.15.173:6791/0 2021-03-09T10:52:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff3500111f0 addr v1:172.21.15.143:6791/0 2021-03-09T10:52:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:52:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500163b0 2021-03-09T10:52:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc90 con 0x7ff3500111f0 2021-03-09T10:52:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:52:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.207+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:52:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.203+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350013100 2021-03-09T10:52:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.207+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:27.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.411+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:52:27.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.411+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:52:27.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.815+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:52:27.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:27.815+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:52:28.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:28.619+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:52:28.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:28.619+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:52:30.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:30.223+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:52:30.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:30.223+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:52:31.863 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:31.859+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:31.879 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:31.875+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:31.908 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:31.907+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:32.065 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:32.063+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:32.066 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:32.063+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013100 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:32.066 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:32.063+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013100 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:32.066 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:32.063+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013100 0x7ff350005390 :-1 s=OPENED pgs=2641 cs=1 l=1).handle_message read tag failed 2021-03-09T10:52:32.066 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:32.063+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013100 0x7ff350005390 :-1 s=OPENED pgs=2641 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:52:32.067 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:32.063+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:52:33.428 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:33.427+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:52:33.428 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:33.427+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:52:36.863 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:36.859+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:36.880 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:36.879+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:36.908 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:36.907+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:37.067 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:37.063+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:39.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:39.831+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:52:39.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:39.831+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:52:41.863 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:41.859+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:41.880 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:41.879+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:41.908 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:41.907+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:42.088 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:42.087+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:46.864 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:46.859+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:46.880 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:46.879+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:46.908 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:46.907+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:47.091 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:47.087+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:51.864 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:51.859+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:51.880 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:51.879+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:51.908 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:51.907+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:52.113 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:52.111+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:52.638 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:52.635+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:52:52.642 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:52.639+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:52:56.864 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:52:56.863+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:52:56.881 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:52:56.879+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:52:56.908 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:52:56.907+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:52:57.115 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:52:57.111+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:52:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:52:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:52:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:52:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350013100 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,7,2] 2021-03-09T10:52:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350005aa0 addr v1:172.21.15.143:6792/0 2021-03-09T10:52:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500170f0 addr v1:172.21.15.173:6792/0 2021-03-09T10:52:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001dab0 addr v1:172.21.15.143:6790/0 2021-03-09T10:52:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001dab0 2021-03-09T10:52:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350005aa0 2021-03-09T10:52:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc90 con 0x7ff3500170f0 2021-03-09T10:52:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.207+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001dab0 legacy=0x7ff35001fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:52:57.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.411+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001dab0 legacy=0x7ff35001fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:52:57.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:57.815+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001dab0 legacy=0x7ff35001fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:52:58.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:52:58.619+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001dab0 legacy=0x7ff35001fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:53:00.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:00.223+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001dab0 legacy=0x7ff35001fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:53:01.864 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:01.863+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:01.865 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.863+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:01.866 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.863+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:01.866 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.863+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 0x7ff35001cba0 :-1 s=OPENED pgs=2697 cs=1 l=1).handle_message read tag failed 2021-03-09T10:53:01.866 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.863+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 0x7ff35001cba0 :-1 s=OPENED pgs=2697 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:53:01.866 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.863+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:53:01.881 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:01.879+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:01.909 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:01.907+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:01.910 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.907+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500170f0 legacy=0x7ff350019510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:01.910 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.907+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500170f0 legacy=0x7ff350019510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:01.911 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.907+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500170f0 0x7ff350019510 :-1 s=OPENED pgs=2653 cs=1 l=1).handle_message read tag failed 2021-03-09T10:53:01.911 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.907+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500170f0 0x7ff350019510 :-1 s=OPENED pgs=2653 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:53:01.911 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.907+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:53:01.911 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:01.907+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:53:02.137 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:02.135+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:03.428 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:03.427+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001dab0 legacy=0x7ff35001fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:53:06.864 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:06.862+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:06.881 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:06.878+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:06.909 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:06.907+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:07.139 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:07.135+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:09.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:09.834+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001dab0 legacy=0x7ff35001fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:53:11.865 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:11.862+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:11.881 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:11.878+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:11.909 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:11.906+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:12.161 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:12.158+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:16.867 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:16.862+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:16.884 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:16.878+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:16.911 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:16.906+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:17.163 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:17.158+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:21.865 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:21.862+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:21.881 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:21.878+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:21.909 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:21.906+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:22.186 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:22.182+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:22.648 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:22.646+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001dab0 legacy=0x7ff35001fea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:53:26.865 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:26.862+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:26.882 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:26.878+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:26.910 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:26.906+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:27.187 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:27.182+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:53:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:53:27.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:53:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500170f0 legacy=0x7ff350019510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350005aa0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001dab0 legacy=0x7ff35001fea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,1,5,7,0,3,2,4,6] 2021-03-09T10:53:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350005aa0 addr v1:172.21.15.143:6793/0 2021-03-09T10:53:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff3500170f0 addr v1:172.21.15.173:6789/0 2021-03-09T10:53:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500163b0 addr v1:172.21.15.173:6791/0 2021-03-09T10:53:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff3500170f0 2021-03-09T10:53:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:53:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc90 con 0x7ff350005aa0 2021-03-09T10:53:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.206+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500170f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:53:27.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.410+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500170f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:53:27.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:27.814+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500170f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:53:28.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:28.618+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500170f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:53:30.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:30.222+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500170f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:53:31.865 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:31.862+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:31.882 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:31.878+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:31.883 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:31.878+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:31.883 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:31.878+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:31.883 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:31.878+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 0x7ff350006a70 :-1 s=OPENED pgs=2665 cs=1 l=1).handle_message read tag failed 2021-03-09T10:53:31.883 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:31.878+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 0x7ff350006a70 :-1 s=OPENED pgs=2665 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:53:31.884 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:31.878+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:53:31.910 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:31.906+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:32.209 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:32.206+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:32.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:32.206+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:53:32.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:32.206+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:32.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:32.206+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350005390 :-1 s=OPENED pgs=2675 cs=1 l=1).handle_message read tag failed 2021-03-09T10:53:32.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:32.206+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff350005390 :-1 s=OPENED pgs=2675 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:53:32.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:32.206+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:53:33.428 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:33.426+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500170f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:53:36.866 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:36.862+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:36.882 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:36.878+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:36.910 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:36.910+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:37.211 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:37.206+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:39.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:39.830+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500170f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:53:41.866 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:41.865+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:41.882 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:41.877+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:41.910 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:41.906+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:42.233 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:42.230+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:46.866 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:46.861+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:46.882 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:46.881+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:46.910 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:46.910+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:47.235 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:47.230+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:51.866 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:51.861+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:51.882 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:51.881+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:51.910 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:51.906+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:52.257 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:52.254+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:52.645 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:52.641+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500170f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:53:56.866 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:53:56.865+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:53:56.883 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:53:56.877+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:53:56.911 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:53:56.905+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:53:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:53:57.209 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:53:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:53:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350005aa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff3500170f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,6,7,2,4,0] 2021-03-09T10:53:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:53:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350015070 addr v1:172.21.15.143:6792/0 2021-03-09T10:53:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001ac00 addr v1:172.21.15.173:6792/0 2021-03-09T10:53:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff3500111f0 2021-03-09T10:53:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350015070 2021-03-09T10:53:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff35001ac00 2021-03-09T10:53:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.205+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:53:57.259 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:53:57.253+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:53:57.411 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.409+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:53:57.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:57.813+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:53:58.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:53:58.617+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:54:00.222 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:00.221+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:54:01.867 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:01.865+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:01.868 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.865+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015070 legacy=0x7ff3500194e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:01.868 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.865+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015070 legacy=0x7ff3500194e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:01.868 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.865+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015070 0x7ff3500194e0 :-1 s=OPENED pgs=2730 cs=1 l=1).handle_message read tag failed 2021-03-09T10:54:01.868 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.865+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015070 0x7ff3500194e0 :-1 s=OPENED pgs=2730 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:54:01.869 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.865+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:54:01.869 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.865+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:54:01.883 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:01.877+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:01.911 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:01.909+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:01.912 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.905+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac00 legacy=0x7ff3500235d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:54:01.912 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.905+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac00 legacy=0x7ff3500235d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:01.912 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.905+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac00 0x7ff3500235d0 :-1 s=OPENED pgs=2686 cs=1 l=1).handle_message read tag failed 2021-03-09T10:54:01.913 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.905+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac00 0x7ff3500235d0 :-1 s=OPENED pgs=2686 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:54:01.913 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:01.905+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:54:02.281 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:02.277+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:03.429 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:03.425+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:54:06.867 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:06.861+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:06.883 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:06.877+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:06.911 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:06.909+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:07.283 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:07.277+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:09.836 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:09.829+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:54:11.867 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:11.865+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:11.883 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:11.877+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:11.911 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:11.909+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:12.305 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:12.301+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:16.867 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:16.861+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:16.883 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:16.881+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:16.912 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:16.905+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:17.307 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:17.301+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:21.867 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:21.864+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:21.883 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:21.880+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:21.912 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:21.909+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:22.329 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:22.325+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:22.649 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:22.648+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:54:26.868 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:26.860+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:26.884 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:26.880+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:26.912 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:26.909+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:54:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:54:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:54:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001ac00 legacy=0x7ff3500235d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015070 legacy=0x7ff3500194e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001cba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,0,2] 2021-03-09T10:54:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350015070 addr v1:172.21.15.143:6791/0 2021-03-09T10:54:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500163b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:54:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff350008520 addr v1:172.21.15.143:6790/0 2021-03-09T10:54:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc90 con 0x7ff3500163b0 2021-03-09T10:54:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350008520 2021-03-09T10:54:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350015070 2021-03-09T10:54:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015070 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:54:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:54:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.204+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350008520 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:27.331 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:27.325+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:27.410 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.408+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:54:27.410 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.408+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015070 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:54:27.410 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.408+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350008520 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:27.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015070 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:54:27.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.812+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:54:27.817 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:27.812+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350008520 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:28.617 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:28.616+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:54:28.618 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:28.616+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015070 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:54:28.618 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:28.616+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350008520 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:30.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:30.220+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015070 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:54:30.225 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:30.220+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:54:30.225 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:30.220+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350008520 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:31.868 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:31.860+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:31.884 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:31.880+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:31.912 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:31.909+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:32.353 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:32.349+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:33.427 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:33.424+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:54:33.428 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:33.424+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015070 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:54:33.428 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:33.424+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350008520 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:36.868 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:36.864+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:36.884 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:36.880+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:36.912 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:36.909+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:37.355 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:37.349+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:39.830 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:39.828+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:54:39.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:39.828+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015070 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:54:39.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:39.828+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350008520 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:41.868 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:41.868+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:41.884 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:41.884+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:41.912 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:41.909+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:42.377 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:42.373+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:46.868 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:46.868+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:46.885 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:46.884+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:46.913 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:46.908+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:47.379 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:47.372+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:51.869 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:51.868+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:51.885 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:51.884+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:51.913 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:51.912+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:52.400 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:52.396+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:52.641 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:52.640+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:54:52.647 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:52.644+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015070 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:54:52.648 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:52.644+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350008520 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:56.869 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:54:56.868+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:54:56.885 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:54:56.884+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:54:56.913 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:54:56.908+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:54:57.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:54:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:54:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:54:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350015070 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff350008520 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:54:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[2,3,6,1,8,7,5,0,4] 2021-03-09T10:54:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:54:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:54:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001cba0 addr v1:172.21.15.143:6792/0 2021-03-09T10:54:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe50 con 0x7ff3500163b0 2021-03-09T10:54:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc90 con 0x7ff3500111f0 2021-03-09T10:54:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cba0 2021-03-09T10:54:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.208+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:54:57.403 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:54:57.396+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:54:57.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.412+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:57.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.412+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:54:57.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.815+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:57.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:57.815+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:54:58.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:58.619+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:54:58.621 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:54:58.619+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:55:00.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:00.223+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:55:00.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:00.223+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:55:01.869 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:01.867+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:01.870 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:01.867+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:01.870 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:01.867+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:01.871 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:01.867+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cba0 0x7ff350005390 :-1 s=OPENED pgs=2757 cs=1 l=1).handle_message read tag failed 2021-03-09T10:55:01.871 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:01.867+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cba0 0x7ff350005390 :-1 s=OPENED pgs=2757 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:55:01.871 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:01.867+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:55:01.885 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:01.883+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:01.913 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:01.912+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:02.425 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:02.420+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:03.428 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:03.427+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:55:03.429 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:03.427+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:55:06.869 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:06.867+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:06.885 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:06.883+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:06.914 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:06.912+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:07.427 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:07.424+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:09.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:09.831+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:55:09.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:09.831+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:55:11.869 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:11.867+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:11.886 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:11.883+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:11.914 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:11.912+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:12.453 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:12.448+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:16.869 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:16.867+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:16.886 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:16.883+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:16.914 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:16.908+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:17.455 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:17.452+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:21.870 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:21.867+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:21.886 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:21.883+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:21.914 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:21.912+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:22.477 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:22.476+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:22.645 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:22.643+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:55:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:22.643+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:55:26.870 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:26.867+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:26.886 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:26.883+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:26.914 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:26.912+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:27.210 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:55:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:55:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:55:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cba0 legacy=0x7ff350005390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,4,0,1,7,8] 2021-03-09T10:55:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350005aa0 addr v1:172.21.15.173:6791/0 2021-03-09T10:55:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350014770 addr v1:172.21.15.143:6791/0 2021-03-09T10:55:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff3500152b0 addr v1:172.21.15.143:6789/0 2021-03-09T10:55:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500152b0 2021-03-09T10:55:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe50 con 0x7ff350014770 2021-03-09T10:55:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001bc90 con 0x7ff350005aa0 2021-03-09T10:55:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014770 legacy=0x7ff350014ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:55:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.207+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500152b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:55:27.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.411+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014770 legacy=0x7ff350014ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:55:27.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.411+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500152b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:55:27.479 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:27.476+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:27.480 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.475+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:27.480 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.475+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:27.481 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.475+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005aa0 0x7ff350014060 :-1 s=OPENED pgs=2740 cs=1 l=1).handle_message read tag failed 2021-03-09T10:55:27.481 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.475+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005aa0 0x7ff350014060 :-1 s=OPENED pgs=2740 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:55:27.481 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.475+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:55:27.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.815+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014770 legacy=0x7ff350014ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:55:27.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:27.815+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500152b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:55:28.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:28.619+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500152b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:55:28.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:28.619+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014770 legacy=0x7ff350014ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:55:30.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:30.223+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014770 legacy=0x7ff350014ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:55:30.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:30.223+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500152b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:55:31.870 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:31.867+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:31.886 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:31.883+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:31.915 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:31.911+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:32.482 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:32.479+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:33.427 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:33.427+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014770 legacy=0x7ff350014ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:55:33.428 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:33.427+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500152b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:55:36.870 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:36.866+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:36.887 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:36.886+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:36.915 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:36.911+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:37.501 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:37.499+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:39.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:39.830+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014770 legacy=0x7ff350014ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:55:39.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:39.830+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500152b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:55:41.870 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:41.866+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:41.887 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:41.882+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:41.915 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:41.911+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:42.503 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:42.499+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:46.871 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:46.866+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:46.887 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:46.886+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:46.915 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:46.911+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:47.525 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:47.523+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:51.870 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:51.866+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:51.887 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:51.882+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:51.915 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:51.915+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:52.527 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:52.523+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:52.647 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:52.646+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014770 legacy=0x7ff350014ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:55:52.648 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:52.646+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500152b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:55:56.871 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:55:56.866+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:55:56.888 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:55:56.882+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:55:56.916 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:55:56.911+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:55:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:55:57.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:55:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:55:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350014770 legacy=0x7ff350014ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff3500152b0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[1,8,7] 2021-03-09T10:55:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:55:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001a820 addr v1:172.21.15.173:6789/0 2021-03-09T10:55:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:55:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500111f0 addr v1:172.21.15.143:6793/0 2021-03-09T10:55:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff3500163b0 addr v1:172.21.15.173:6792/0 2021-03-09T10:55:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001a820 2021-03-09T10:55:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:55:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe50 con 0x7ff3500111f0 2021-03-09T10:55:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:57.218 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.206+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:55:57.413 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.410+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:55:57.549 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:55:57.547+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:55:57.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:57.814+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:55:58.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:55:58.618+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:00.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:00.222+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:01.871 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:01.866+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:01.887 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:01.882+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:01.888 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.882+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:01.888 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.882+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.882+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=2734 cs=1 l=1).handle_message read tag failed 2021-03-09T10:56:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.882+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=2734 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:56:01.889 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.882+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:56:01.916 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:01.915+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:01.916 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.914+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:56:01.917 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.914+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:01.917 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.914+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff35001c490 :-1 s=OPENED pgs=2746 cs=1 l=1).handle_message read tag failed 2021-03-09T10:56:01.917 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.914+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 0x7ff35001c490 :-1 s=OPENED pgs=2746 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:56:01.917 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:01.914+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:56:02.551 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:02.547+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:03.428 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:03.426+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:06.871 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:06.866+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:06.888 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:06.886+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:06.916 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:06.915+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:07.573 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:07.571+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:09.835 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:09.834+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:11.871 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:11.866+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:11.888 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:11.886+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:11.916 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:11.915+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:12.575 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:12.571+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:16.872 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:16.865+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:16.888 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:16.885+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:16.916 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:16.915+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:17.597 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:17.595+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:21.872 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:21.869+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:21.888 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:21.885+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:21.916 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:21.914+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:22.599 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:22.594+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:22.648 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:22.645+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:26.872 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:26.869+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:26.888 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:26.885+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:26.917 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:26.914+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:27.211 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:56:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:56:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:56:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[8,2,1,7,3,4,6,0,5] 2021-03-09T10:56:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff3500111f0 addr v1:172.21.15.143:6793/0 2021-03-09T10:56:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff35001cba0 addr v1:172.21.15.143:6790/0 2021-03-09T10:56:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff35001a820 addr v1:172.21.15.173:6789/0 2021-03-09T10:56:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001a820 2021-03-09T10:56:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cba0 2021-03-09T10:56:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:56:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cba0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:56:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.209+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:27.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.413+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cba0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:56:27.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.413+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:27.620 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:27.618+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:27.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.817+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cba0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:56:27.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:27.817+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:28.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:28.621+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cba0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:56:28.625 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:28.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:30.226 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:30.225+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cba0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:56:30.226 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:30.225+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:31.872 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:31.869+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:31.889 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:31.885+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:31.890 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:31.885+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:31.891 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:31.885+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:31.891 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:31.885+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=2747 cs=1 l=1).handle_message read tag failed 2021-03-09T10:56:31.892 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:31.885+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 0x7ff350006a70 :-1 s=OPENED pgs=2747 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:56:31.892 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:31.885+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:56:31.917 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:31.914+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:32.623 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:32.618+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:33.429 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:33.429+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cba0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:56:33.430 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:33.429+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:36.872 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:36.869+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:36.889 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:36.885+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:36.921 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:36.918+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:37.645 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:37.642+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:39.836 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:39.833+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cba0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:56:39.837 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:39.833+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:41.872 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:41.869+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:41.889 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:41.889+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:41.921 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:41.918+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:42.647 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:42.642+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:46.873 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:46.869+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:46.890 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:46.885+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:46.922 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:46.918+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:47.669 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:47.666+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:51.873 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:51.868+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:51.889 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:51.888+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:51.922 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:51.918+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:52.646 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:52.644+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cba0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:56:52.650 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:52.644+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:56:52.671 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:52.666+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:56.873 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:56:56.868+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:56:56.890 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:56:56.888+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:56:56.922 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:56:56.918+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:56:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:56:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:56:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:56:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff3500111f0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff35001cba0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff35001a820 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[6,0,4,7,3,5] 2021-03-09T10:56:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff35001cfa0 addr v1:172.21.15.143:6792/0 2021-03-09T10:56:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff350014750 addr v1:172.21.15.143:6789/0 2021-03-09T10:56:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:56:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe50 con 0x7ff350014750 2021-03-09T10:56:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350005aa0 2021-03-09T10:56:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cfa0 2021-03-09T10:56:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350014750 legacy=0x7ff350014b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:56:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.208+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:56:57.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.412+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350014750 legacy=0x7ff350014b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:56:57.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.412+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:56:57.693 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:56:57.690+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:56:57.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.816+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:56:57.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:57.816+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350014750 legacy=0x7ff350014b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:56:58.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:58.620+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350014750 legacy=0x7ff350014b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:56:58.625 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:56:58.620+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:57:00.226 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:00.224+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350014750 legacy=0x7ff350014b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:57:00.227 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:00.224+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:57:01.873 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:01.868+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:01.874 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:01.868+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfa0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:01.874 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:01.868+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfa0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:01.875 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:01.868+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfa0 0x7ff35001a110 :-1 s=OPENED pgs=2822 cs=1 l=1).handle_message read tag failed 2021-03-09T10:57:01.875 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:01.868+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfa0 0x7ff35001a110 :-1 s=OPENED pgs=2822 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:57:01.875 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:01.868+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:57:01.875 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:01.868+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:57:01.890 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:01.884+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:01.922 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:01.918+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:02.695 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:02.690+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:03.432 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:03.428+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:57:03.432 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:03.428+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350014750 legacy=0x7ff350014b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:57:06.873 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:06.868+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:06.890 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:06.884+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:06.922 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:06.918+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:07.717 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:07.713+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:09.833 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:09.832+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350014750 legacy=0x7ff350014b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:57:09.834 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:09.832+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:57:11.874 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:11.868+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:11.890 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:11.884+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:11.922 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:11.917+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:12.719 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:12.713+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:16.874 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:16.872+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:16.890 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:16.884+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:16.922 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:16.921+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:17.741 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:17.737+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:21.874 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:21.872+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:21.891 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:21.888+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:21.923 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:21.917+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:22.644+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:57:22.647 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:22.644+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350014750 legacy=0x7ff350014b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:57:22.743 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:22.737+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:26.874 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:26.872+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:26.891 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:26.888+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:26.923 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:26.921+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:57:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:57:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:57:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff35001cfa0 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350023af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff350014750 legacy=0x7ff350014b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[7,5,3] 2021-03-09T10:57:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001cfa0 addr v1:172.21.15.173:6792/0 2021-03-09T10:57:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff3500163b0 addr v1:172.21.15.173:6791/0 2021-03-09T10:57:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500111f0 addr v1:172.21.15.173:6790/0 2021-03-09T10:57:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500111f0 2021-03-09T10:57:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe50 con 0x7ff3500163b0 2021-03-09T10:57:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001cfa0 2021-03-09T10:57:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:27.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.208+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:27.412 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.412+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:27.764 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:27.761+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:27.765 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.764+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:27.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.764+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:27.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.764+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff35001c490 :-1 s=OPENED pgs=2803 cs=1 l=1).handle_message read tag failed 2021-03-09T10:57:27.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.764+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 0x7ff35001c490 :-1 s=OPENED pgs=2803 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:57:27.766 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.764+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:57:27.816 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:27.816+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:28.620 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:28.620+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:30.224 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:30.223+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:31.874 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:31.871+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:31.891 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:31.887+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:31.923 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:31.917+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:31.924 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:31.919+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:31.925 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:31.919+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:31.925 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:31.919+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfa0 0x7ff350006a70 :-1 s=OPENED pgs=2793 cs=1 l=1).handle_message read tag failed 2021-03-09T10:57:31.925 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:31.919+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfa0 0x7ff350006a70 :-1 s=OPENED pgs=2793 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:57:31.925 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:31.919+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:57:32.767 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:32.761+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:33.429 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:33.427+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:36.875 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:36.871+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:36.891 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:36.887+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:36.924 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:36.921+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:37.789 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:37.785+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:39.832 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:39.831+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:41.875 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:41.871+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:41.891 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:41.887+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:41.923 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:41.917+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:42.791 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:42.785+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:46.875 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:46.871+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:46.892 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:46.887+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:46.924 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:46.921+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:47.813 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:47.809+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:51.875 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:51.871+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:51.892 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:51.891+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:51.924 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:51.921+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:52.636 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:52.635+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:52.815 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:52.809+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:56.875 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:57:56.871+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:57:56.892 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:57:56.891+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:57:56.924 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:57:56.920+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:57:57.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:57:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:57:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:57:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001cfa0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff3500163b0 legacy=0x7ff35001c490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500111f0 legacy=0x7ff350003380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:57:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[3,7,0,8,2,5,1,6,4] 2021-03-09T10:57:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 10 monclient(hunting): picked mon.g con 0x7ff3500163b0 addr v1:172.21.15.173:6790/0 2021-03-09T10:57:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 10 monclient(hunting): picked mon.i con 0x7ff35001c490 addr v1:172.21.15.173:6792/0 2021-03-09T10:57:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 10 monclient(hunting): picked mon.a con 0x7ff35001cf80 addr v1:172.21.15.143:6789/0 2021-03-09T10:57:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff35001cf80 2021-03-09T10:57:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff3500163b0 2021-03-09T10:57:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff35001c490 2021-03-09T10:57:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:57:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.211+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:57.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.415+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:57:57.417 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.415+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:57.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.819+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:57:57.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:57.819+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:57:57.836 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:57:57.832+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:57:58.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:58.623+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:57:58.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:57:58.623+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:58:00.228 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:00.227+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:58:00.229 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:00.227+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:58:01.876 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:01.871+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:01.892 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:01.891+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:01.924 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:01.920+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:01.925 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:01.923+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:01.925 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:01.923+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:01.926 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:01.923+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c490 0x7ff35001c870 :-1 s=OPENED pgs=2808 cs=1 l=1).handle_message read tag failed 2021-03-09T10:58:01.926 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:01.923+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c490 0x7ff35001c870 :-1 s=OPENED pgs=2808 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:58:01.926 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:01.923+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6792/0 2021-03-09T10:58:02.839 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:02.836+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:03.432 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:03.431+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:58:03.432 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:03.431+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:58:06.876 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:06.875+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:06.892 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:06.891+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:06.925 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:06.924+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:07.861 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:07.856+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:09.839 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:09.834+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:58:09.839 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:09.834+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:58:11.876 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:11.874+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:11.893 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:11.890+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:11.925 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:11.924+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:12.863 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:12.856+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:16.876 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:16.874+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:16.893 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:16.890+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:16.925 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:16.924+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:17.885 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:17.880+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:21.876 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:21.874+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:21.893 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:21.890+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:21.925 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:21.924+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:22.652 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:22.650+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6789/0 2021-03-09T10:58:22.652 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:22.650+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6790/0 2021-03-09T10:58:22.887 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:22.880+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:26.876 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:26.874+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:26.893 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:26.890+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:26.925 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:26.920+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:27.212 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:58:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:58:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:58:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6792/0 conn(0x7ff35001c490 legacy=0x7ff35001c870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:27.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6789/0 conn(0x7ff35001cf80 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[4,8,6,1,5,2] 2021-03-09T10:58:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 10 monclient(hunting): picked mon.c con 0x7ff350005aa0 addr v1:172.21.15.143:6791/0 2021-03-09T10:58:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 10 monclient(hunting): picked mon.e con 0x7ff350014750 addr v1:172.21.15.143:6793/0 2021-03-09T10:58:27.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 10 monclient(hunting): picked mon.d con 0x7ff350015270 addr v1:172.21.15.143:6792/0 2021-03-09T10:58:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff350005aa0 2021-03-09T10:58:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6792/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350015270 2021-03-09T10:58:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6793/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350007180 con 0x7ff350014750 2021-03-09T10:58:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:27.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.210+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:58:27.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.414+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:58:27.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:27.818+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:58:27.889 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:27.884+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:28.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:28.622+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:58:30.228 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:30.226+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:58:31.877 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.874+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015270 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:31.877 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.874+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015270 legacy=0x7ff35001a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:31.877 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.874+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015270 0x7ff35001a110 :-1 s=OPENED pgs=2881 cs=1 l=1).handle_message read tag failed 2021-03-09T10:58:31.878 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.874+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015270 0x7ff35001a110 :-1 s=OPENED pgs=2881 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:58:31.878 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.874+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6792/0 2021-03-09T10:58:31.878 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:31.874+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:31.893 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:31.890+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:31.894 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.890+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014750 legacy=0x7ff350014b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:31.895 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.890+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014750 legacy=0x7ff350014b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:31.895 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.890+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014750 0x7ff350014b60 :-1 s=OPENED pgs=2816 cs=1 l=1).handle_message read tag failed 2021-03-09T10:58:31.895 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.890+0000 7ff373fff700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014750 0x7ff350014b60 :-1 s=OPENED pgs=2816 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:58:31.895 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.890+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 reap_dead start 2021-03-09T10:58:31.895 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:31.890+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6793/0 2021-03-09T10:58:31.925 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:31.924+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:32.909 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:32.904+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:33.430 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:33.430+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:58:36.877 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:36.874+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:36.893 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:36.890+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:36.925 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:36.924+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:37.911 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:37.908+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:39.837 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:39.834+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:58:41.877 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:41.874+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:41.894 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:41.894+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:41.926 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:41.924+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:42.932 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:42.932+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:46.877 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:46.873+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:46.894 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:46.889+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:46.926 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:46.923+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:47.935 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:47.931+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:51.877 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:51.873+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:51.894 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:51.889+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:51.927 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:51.923+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:52.650 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:52.649+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6791/0 2021-03-09T10:58:52.956 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:52.955+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:56.877 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:58:56.877+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:58:56.894 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:58:56.889+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:58:56.926 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:58:56.923+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:58:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 10 monclient(hunting): tick 2021-03-09T10:58:57.213 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 1 monclient(hunting): continuing hunt 2021-03-09T10:58:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:58:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6793/0 conn(0x7ff350014750 legacy=0x7ff350014b60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6792/0 conn(0x7ff350015270 legacy=0x7ff35001a110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:57.214 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6791/0 conn(0x7ff350005aa0 legacy=0x7ff350014040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 10 monclient: _add_conns ranks=[5,1,2] 2021-03-09T10:58:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 10 monclient(hunting): picked mon.h con 0x7ff350015270 addr v1:172.21.15.173:6791/0 2021-03-09T10:58:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 10 monclient(hunting): picked mon.f con 0x7ff350014750 addr v1:172.21.15.173:6789/0 2021-03-09T10:58:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 10 monclient(hunting): picked mon.b con 0x7ff3500163b0 addr v1:172.21.15.143:6790/0 2021-03-09T10:58:57.215 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff350016040 con 0x7ff350014750 2021-03-09T10:58:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35000fe10 con 0x7ff3500163b0 2021-03-09T10:58:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 1 -- 172.21.15.143:0/2492024602 --> v1:172.21.15.173:6791/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7ff35001b560 con 0x7ff350015270 2021-03-09T10:58:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff35e7fc700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:57.216 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:58:57.217 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.209+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014750 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:58:57.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.413+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:58:57.416 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.413+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014750 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:58:57.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.817+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:58:57.820 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.817+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014750 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:58:57.959 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:58:57.955+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:58:57.960 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.953+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015270 legacy=0x7ff350014b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:57.960 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.953+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015270 legacy=0x7ff350014b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:57.960 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.953+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015270 0x7ff350014b60 :-1 s=OPENED pgs=2858 cs=1 l=1).handle_message read tag failed 2021-03-09T10:58:57.960 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.953+0000 7ff3737fe700 1 --1- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6791/0 conn(0x7ff350015270 0x7ff350014b60 :-1 s=OPENED pgs=2858 cs=1 l=1).fault on lossy channel, failing 2021-03-09T10:58:57.961 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:57.953+0000 7ff35f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.173:6791/0 2021-03-09T10:58:58.624 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:58.621+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:58:58.625 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:58:58.621+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014750 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:59:00.226 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:59:00.225+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:59:00.226 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:59:00.225+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014750 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:59:01.878 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:59:01.877+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:59:01.895 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:59:01.889+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:59:01.929 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:59:01.923+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:59:02.981 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:59:02.979+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:59:03.432 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:59:03.429+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:59:03.432 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:59:03.429+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014750 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:59:06.878 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:59:06.877+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:59:06.895 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:59:06.889+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:59:06.927 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:59:06.923+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:59:07.983 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:59:07.979+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:59:09.839 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:59:09.837+0000 7ff3737fe700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.143:6790/0 conn(0x7ff3500163b0 legacy=0x7ff350005390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.143:6790/0 2021-03-09T10:59:09.839 INFO:teuthology.orchestra.run.smithi143.stderr:2021-03-09T10:59:09.837+0000 7ff373fff700 1 -- 172.21.15.143:0/2492024602 >> v1:172.21.15.173:6789/0 conn(0x7ff350014750 legacy=0x7ff350006a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.173:6789/0 2021-03-09T10:59:11.878 INFO:tasks.ceph.mon.d.smithi143.stderr:2021-03-09T10:59:11.877+0000 7f445d90b700 -1 mon.d@6(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 3 at 2021-03-09T09:35:41.143806+0000) 2021-03-09T10:59:11.895 INFO:tasks.ceph.mon.e.smithi143.stderr:2021-03-09T10:59:11.889+0000 7fd6e9553700 -1 mon.e@8(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.143990+0000) 2021-03-09T10:59:11.927 INFO:tasks.ceph.mon.i.smithi173.stderr:2021-03-09T10:59:11.923+0000 7fbab2ccd700 -1 mon.i@7(electing) e1 get_health_metrics reporting 6 slow ops, oldest is log(1 entries from seq 5 at 2021-03-09T09:35:41.149227+0000) 2021-03-09T10:59:13.004 INFO:tasks.ceph.mon.h.smithi173.stderr:2021-03-09T10:59:13.003+0000 7fcabb99b700 -1 mon.h@5(electing) e1 get_health_metrics reporting 9 slow ops, oldest is log(1 entries from seq 7 at 2021-03-09T09:35:32.463973+0000) 2021-03-09T10:59:15.647 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-09T10:59:15.648 DEBUG:teuthology.task.console_log:Killing console logger for smithi143 2021-03-09T10:59:15.649 DEBUG:teuthology.task.console_log:Killing console logger for smithi173 2021-03-09T10:59:15.649 DEBUG:teuthology.exit:Finished running handlers