2019-10-04T11:02:17.539 INFO:root:teuthology version: 1.0.0-b6cc51f 2019-10-04T11:02:17.539 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.fDpiTq.tmp -- --verbose --lock --description fs/multiclient/{begin.yaml clusters/1-mds-3-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distros/ubuntu_latest.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/cephfs_misc_tests.yaml} --name pdonnell-2019-10-04_05:23:55-fs-wip-pdonnell-testing-20191004.015615-distro-basic-smithi --block --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2019-10-04_05:23:55-fs-wip-pdonnell-testing-20191004.015615-distro-basic-smithi/4356997 2019-10-04T11:02:17.580 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-04T11:02:17.634 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2019-10-04_05:23:55-fs-wip-pdonnell-testing-20191004.015615-distro-basic-smithi/4356997 branch: wip-pdonnell-testing-20191004.015615 description: fs/multiclient/{begin.yaml clusters/1-mds-3-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distros/ubuntu_latest.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/cephfs_misc_tests.yaml} email: pdonnell@redhat.com first_in_suite: false job_id: '4356997' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2019-10-04_05:23:55-fs-wip-pdonnell-testing-20191004.015615-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-pdonnell-testing-20191004.015615 ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - evicting unresponsive client - POOL_APP_NOT_ENABLED - has not responded to cap revoke by MDS for over sha1: bacb377f1e7bd9f809978f6ebeb05ff4d801b875 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: bacb377f1e7bd9f809978f6ebeb05ff4d801b875 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20191004.015615 sha1: bacb377f1e7bd9f809978f6ebeb05ff4d801b875 owner: scheduled_pdonnell@teuthology priority: 95 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - - client.1 - - client.2 sha1: bacb377f1e7bd9f809978f6ebeb05ff4d801b875 suite: fs suite_branch: wip-pdonnell-testing-20191004.015615 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191004.015615/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: bacb377f1e7bd9f809978f6ebeb05ff4d801b875 tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_misc teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.17289 2019-10-04T11:02:17.635 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191004.015615/qa; will attempt to use it 2019-10-04T11:02:17.635 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191004.015615/qa/tasks 2019-10-04T11:02:17.635 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-10-04T11:02:17.637 INFO:teuthology.task.internal:Checking packages... 2019-10-04T11:02:17.649 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'bacb377f1e7bd9f809978f6ebeb05ff4d801b875' 2019-10-04T11:02:17.649 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-10-04T11:02:17.649 INFO:teuthology.packaging:ref: None 2019-10-04T11:02:17.649 INFO:teuthology.packaging:tag: None 2019-10-04T11:02:17.650 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20191004.015615 2019-10-04T11:02:17.650 INFO:teuthology.packaging:sha1: bacb377f1e7bd9f809978f6ebeb05ff4d801b875 2019-10-04T11:02:17.650 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-pdonnell-testing-20191004.015615 2019-10-04T11:02:17.826 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-5761-gbacb377-1bionic 2019-10-04T11:02:17.829 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-10-04T11:02:17.839 INFO:teuthology.task.internal:no buildpackages task found 2019-10-04T11:02:17.839 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-10-04T11:02:17.856 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-10-04T11:02:17.856 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-04T11:02:17.908 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:02:27.924 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:02:37.941 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:02:47.956 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:02:57.967 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:03:07.983 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:03:17.999 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:03:28.012 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:03:38.028 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:03:48.053 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:03:58.068 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:04:08.084 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:04:18.099 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:04:28.112 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:04:38.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:04:48.144 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:04:58.160 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:05:08.177 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:05:18.191 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:05:28.209 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:05:38.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:05:48.235 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:05:58.252 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:06:08.263 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:06:18.274 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:06:28.289 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:06:38.304 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:06:48.320 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:06:58.328 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:07:08.343 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:07:18.357 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:07:28.373 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:07:38.388 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:07:48.405 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:07:58.422 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:08:08.440 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:08:18.457 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:08:28.482 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:08:38.492 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:08:48.503 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:08:58.519 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:09:08.536 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:09:18.552 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:09:28.569 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:09:38.586 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:09:48.601 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:09:58.616 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:10:08.635 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:10:18.652 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:10:28.669 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:10:38.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:10:48.695 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:10:58.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:11:08.730 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:11:18.747 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:11:28.760 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:11:38.776 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:11:48.785 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:11:58.800 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:12:08.816 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:12:18.830 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:12:28.844 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:12:38.852 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:12:48.870 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:12:58.887 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:13:08.897 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:13:18.913 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:13:28.925 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:13:38.941 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:13:48.956 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:13:58.972 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:14:08.989 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:14:18.998 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:14:29.011 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:14:39.027 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:14:49.043 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:14:59.059 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:15:09.073 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:15:19.082 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:15:29.101 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:15:39.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:15:49.133 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 4)... 2019-10-04T11:15:59.151 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 4)... 2019-10-04T11:16:09.166 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:16:19.180 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:16:29.197 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:16:39.213 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:16:49.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:16:59.245 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:17:09.262 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:17:19.279 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:17:29.296 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:17:39.314 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:17:49.323 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:17:59.339 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:18:09.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:18:19.591 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:18:29.608 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:18:39.619 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 1)... 2019-10-04T11:18:49.636 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:18:59.645 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:19:09.662 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:19:19.679 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:19:29.692 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:19:39.708 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:19:49.725 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:19:59.734 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 5)... 2019-10-04T11:20:09.751 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:20:19.768 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:20:29.777 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 1)... 2019-10-04T11:20:39.788 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 1)... 2019-10-04T11:20:49.800 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:20:59.817 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:21:09.835 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:21:19.853 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 3)... 2019-10-04T11:21:29.868 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 5)... 2019-10-04T11:21:39.880 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:21:49.897 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:21:59.913 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:22:09.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:22:19.940 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:22:29.952 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:22:39.970 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:22:49.981 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:22:59.998 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:23:10.014 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:23:20.030 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:23:30.046 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:23:40.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:23:50.078 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:24:00.095 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:24:10.112 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:24:20.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:24:30.144 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:24:40.156 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:24:50.174 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:25:00.190 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:25:10.210 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:25:20.224 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:25:30.241 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:25:40.260 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:25:50.279 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:26:00.529 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 4)... 2019-10-04T11:26:10.542 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:26:20.558 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 0)... 2019-10-04T11:26:30.568 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:26:40.576 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:26:50.585 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:27:00.599 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:27:10.615 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:27:20.628 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:27:30.644 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:27:40.660 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:27:50.672 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:28:00.688 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:28:10.704 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:28:20.720 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:28:30.736 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:28:40.748 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:28:50.765 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:29:00.780 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:29:10.797 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:29:20.809 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:29:30.827 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:29:40.840 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:29:50.856 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:30:00.873 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:30:10.891 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:30:20.904 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:30:30.914 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:30:40.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:30:50.945 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:31:00.960 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:31:10.976 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:31:20.993 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:31:31.009 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:31:41.028 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:31:51.046 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:32:01.063 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:32:11.081 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:32:21.102 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:32:31.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:32:41.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:32:51.148 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:33:01.163 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:33:11.180 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 4)... 2019-10-04T11:33:21.193 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:33:31.208 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:33:41.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:33:51.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:34:01.259 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:34:11.273 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:34:21.290 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:34:31.306 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:34:41.323 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:34:51.336 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:35:01.353 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:35:11.372 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:35:21.388 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:35:31.403 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:35:41.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:35:51.438 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:36:01.456 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:36:11.466 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:36:21.481 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:36:31.496 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:36:41.513 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:36:51.531 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:37:01.548 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:37:11.560 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:37:21.577 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:37:31.591 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:37:41.601 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:37:51.617 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:38:01.633 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:38:11.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:38:21.659 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:38:31.678 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:38:41.692 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:38:51.700 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:39:01.717 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:39:11.732 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:39:21.748 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:39:31.757 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:39:41.767 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:39:51.784 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)... 2019-10-04T11:40:01.802 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 5, have 2)...