2020-05-18T17:54:33.068 INFO:root:teuthology version: 1.0.0-437df57 2020-05-18T17:54:33.070 DEBUG:teuthology.run:Teuthology command: teuthology --name jlayton-2020-05-18_17:49:53-kcephfs-octopus-distro-basic-smithi --owner scheduled_jlayton@teuthology --lock --verbose --description kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/client-limits.yaml} --block --archive /home/teuthworker/archive/jlayton-2020-05-18_17:49:53-kcephfs-octopus-distro-basic-smithi/5067858 -- /tmp/teuthology-worker.ph_8nu1f.tmp 2020-05-18T17:54:33.111 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-05-18T17:54:33.186 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/jlayton-2020-05-18_17:49:53-kcephfs-octopus-distro-basic-smithi/5067858 branch: octopus description: kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/client-limits.yaml} email: jlayton@redhat.com first_in_suite: false job_id: '5067858' kernel: client: branch: testing kdb: true rpm: /home/jlayton/rhel8.3/kernel-4.18.0.bz1827767.003+-2.x86_64.rpm sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: jlayton-2020-05-18_17:49:53-kcephfs-octopus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: octopus ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - responding to mclientcaps\(revoke\) - not advance its oldest_client_tid - failing to advance its oldest client/flush tid - Too many inodes in cache - failing to respond to cache pressure - slow requests are blocked - failing to respond to capability release - MDS cache is too large - \(MDS_CLIENT_OLDEST_TID\) - \(MDS_CACHE_OVERSIZED\) sha1: f2cf2ce1bd9a86462510a7a12afa4e528b615df2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: f2cf2ce1bd9a86462510a7a12afa4e528b615df2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: master sha1: ab2488773626d40fb1f8345a99f4bbd6712c0fd4 owner: scheduled_jlayton@teuthology priority: 100 repo: git://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.b - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.c - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - - client.1 - - client.2 - - client.3 sha1: f2cf2ce1bd9a86462510a7a12afa4e528b615df2 sleep_before_teardown: 0 suite: kcephfs suite_branch: master suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_master/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: ab2488773626d40fb1f8345a99f4bbd6712c0fd4 tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - dump - indent rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_client_limits teuthology_branch: master tube: smithi verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.20687 2020-05-18T17:54:33.186 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_master/qa; will attempt to use it 2020-05-18T17:54:33.187 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_master/qa/tasks 2020-05-18T17:54:33.188 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-05-18T17:54:33.190 INFO:teuthology.task.internal:Checking packages... 2020-05-18T17:54:33.227 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'f2cf2ce1bd9a86462510a7a12afa4e528b615df2' 2020-05-18T17:54:33.227 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-05-18T17:54:33.228 INFO:teuthology.packaging:ref: None 2020-05-18T17:54:33.229 INFO:teuthology.packaging:tag: None 2020-05-18T17:54:33.229 INFO:teuthology.packaging:branch: octopus 2020-05-18T17:54:33.229 INFO:teuthology.packaging:sha1: f2cf2ce1bd9a86462510a7a12afa4e528b615df2 2020-05-18T17:54:33.230 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2020-05-18T17:54:33.735 INFO:teuthology.task.internal:Found packages for ceph version 15.2.2-1bionic 2020-05-18T17:54:33.735 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-05-18T17:54:33.768 INFO:teuthology.task.internal:no buildpackages task found 2020-05-18T17:54:33.769 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-05-18T17:54:33.820 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-05-18T17:54:33.821 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-05-18T17:54:33.885 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:54:43.905 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:54:53.923 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:55:03.939 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:55:13.958 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:55:23.980 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:55:33.995 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:55:44.007 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:55:54.021 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:56:04.039 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:56:14.054 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:56:24.071 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:56:34.089 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:56:44.108 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:56:54.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:57:04.152 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:57:14.172 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:57:24.200 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:57:34.221 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:57:44.237 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:57:54.255 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:58:04.270 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:58:14.292 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:58:24.311 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:58:34.326 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:58:44.343 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:58:54.359 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:59:04.376 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T17:59:14.396 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:59:24.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:59:34.435 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:59:44.448 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T17:59:54.471 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:00:04.488 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:00:14.502 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:00:24.523 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:00:34.541 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:00:44.556 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:00:54.571 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:01:04.583 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:01:14.601 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:01:24.615 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:01:34.632 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:01:44.653 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:01:54.678 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:02:04.703 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:02:14.720 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:02:24.732 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:02:34.747 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:02:44.768 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:02:54.786 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:03:04.806 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:03:14.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:03:24.844 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:03:34.870 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:03:44.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:03:54.899 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:04:04.917 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:04:14.935 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:04:24.950 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:04:34.967 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:04:44.988 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:04:55.008 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:05:05.032 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:05:15.051 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:05:25.070 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:05:35.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:05:45.110 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:05:55.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:06:05.148 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:06:15.166 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:06:25.182 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:06:35.202 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:06:45.212 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:06:55.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:07:05.247 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:07:15.268 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:07:25.288 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:07:35.303 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:07:45.320 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:07:55.335 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:08:05.352 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:08:15.372 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:08:25.392 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:08:35.408 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:08:45.428 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:08:55.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:09:05.497 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:09:15.522 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:09:25.539 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:09:35.558 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:09:45.576 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:09:55.616 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:10:05.633 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:10:15.653 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:10:25.672 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:10:35.694 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:10:45.732 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:10:55.764 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:11:05.784 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:11:15.802 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:11:25.825 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:11:35.845 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:11:45.866 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:11:55.880 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:12:05.896 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:12:15.916 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:12:25.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:12:35.949 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:12:45.969 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:12:55.991 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:13:06.004 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:13:16.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:13:26.045 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:13:36.056 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:13:46.076 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:13:56.092 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:14:06.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:14:16.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:14:26.146 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:14:36.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:14:46.174 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:14:56.191 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:15:06.202 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:15:16.219 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:15:26.236 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:15:36.246 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:15:46.263 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:15:56.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:16:06.321 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:16:16.343 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:16:26.365 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:16:36.408 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:16:46.432 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:16:56.452 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:17:06.473 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:17:16.497 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:17:26.516 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:17:36.542 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:17:46.558 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:17:56.571 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:18:06.587 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:18:16.606 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:18:26.626 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:18:36.643 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:18:46.659 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:18:56.676 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:19:06.696 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:19:16.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:19:26.747 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:19:36.765 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:19:46.783 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:19:56.800 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:20:06.819 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:20:16.837 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:20:26.856 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:20:36.871 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:20:46.887 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:20:56.908 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:21:06.933 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:21:16.951 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:21:26.963 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:21:36.977 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:21:46.996 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:21:57.016 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:22:07.031 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:22:17.050 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:22:27.072 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:22:37.091 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:22:47.110 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:22:57.126 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:23:07.147 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:23:17.163 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:23:27.181 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:23:37.200 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:23:47.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:23:57.246 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:24:07.262 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:24:17.278 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:24:27.294 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:24:37.314 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:24:47.335 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:24:57.361 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:25:07.385 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:25:17.407 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:25:27.427 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:25:37.451 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:25:47.475 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:25:57.491 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:26:07.515 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:26:17.533 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:26:27.551 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:26:37.575 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:26:47.591 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:26:57.608 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:27:07.628 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:27:17.646 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:27:27.681 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:27:37.695 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:27:47.711 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:27:57.727 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:28:07.746 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:28:17.765 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:28:27.788 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:28:37.808 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:28:47.835 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:28:57.852 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:29:07.875 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:29:17.896 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:29:27.927 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:29:37.945 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-05-18T18:29:47.964 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:29:57.984 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:30:08.004 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:30:18.022 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:30:28.039 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:30:38.059 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:30:48.079 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:30:58.100 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-05-18T18:31:08.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:31:18.137 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:31:28.156 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:31:38.194 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:31:48.216 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:31:58.235 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:32:08.255 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:32:18.273 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:32:28.294 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-05-18T18:32:38.310 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)...