2020-11-16T21:40:24.516 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-16T21:40:24.518 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi/5631931 --owner scheduled_pdonnell@teuthology --description multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_multimds_misc} --verbose --block --name pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi --lock -- /tmp/teuthology-worker.807ep1v_.tmp 2020-11-16T21:40:24.560 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T21:40:24.623 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi/5631931 branch: wip-pdonnell-testing-20201116.174825 description: multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/3-mds conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{basic/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_multimds_misc} email: pdonnell@redhat.com first_in_suite: false job_id: '5631931' kernel: client: branch: testing kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-pdonnell-testing-20201116.174825 ceph: cephfs: max_mds: 3 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 fuse default permissions: false rados mon op timeout: 15m rados osd op timeout: 15m global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Scrub error on inode log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 6959059e6aae2fb5241382a4eeeca62785867604 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 6959059e6aae2fb5241382a4eeeca62785867604 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20201116.174825 sha1: 6959059e6aae2fb5241382a4eeeca62785867604 owner: scheduled_pdonnell@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mgr.x - mds.b - mds.c - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - client.1 sha1: 6959059e6aae2fb5241382a4eeeca62785867604 sleep_before_teardown: 0 suite: multimds suite_branch: wip-pdonnell-testing-20201116.174825 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201116.174825/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 6959059e6aae2fb5241382a4eeeca62785867604 tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - check-counter: counters: mds: - mds.exported - mds.imported - cephfs_test_runner: modules: - tasks.cephfs.test_multimds_misc teuthology_branch: master timestamp: 2020-11-16_20:59:43 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.7054 2020-11-16T21:40:24.624 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201116.174825/qa; will attempt to use it 2020-11-16T21:40:24.624 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201116.174825/qa/tasks 2020-11-16T21:40:24.625 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-16T21:40:24.626 INFO:teuthology.task.internal:Checking packages... 2020-11-16T21:40:24.645 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '6959059e6aae2fb5241382a4eeeca62785867604' 2020-11-16T21:40:24.646 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-16T21:40:24.646 INFO:teuthology.packaging:ref: None 2020-11-16T21:40:24.646 INFO:teuthology.packaging:tag: None 2020-11-16T21:40:24.647 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20201116.174825 2020-11-16T21:40:24.647 INFO:teuthology.packaging:sha1: 6959059e6aae2fb5241382a4eeeca62785867604 2020-11-16T21:40:24.647 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20201116.174825 2020-11-16T21:40:24.808 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7226.g6959059e 2020-11-16T21:40:24.808 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-16T21:40:24.817 INFO:teuthology.task.internal:no buildpackages task found 2020-11-16T21:40:24.817 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-16T21:40:24.831 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-16T21:40:24.832 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-16T21:40:24.880 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:40:34.892 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:40:44.911 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:40:54.925 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:41:04.943 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:41:14.962 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:41:24.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:41:35.004 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:41:45.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:41:55.051 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:42:05.068 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:42:15.086 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:42:25.097 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:42:35.115 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:42:45.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:42:55.150 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:43:05.159 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:43:15.178 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:43:25.194 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:43:35.211 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:43:45.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:43:55.239 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:44:05.259 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:44:15.276 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:44:25.287 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:44:35.305 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:44:45.317 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:44:55.336 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:45:05.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:45:15.377 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:45:25.394 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:45:35.412 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:45:45.422 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:45:55.441 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:46:05.459 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:46:15.470 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:46:25.488 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:46:35.499 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:46:45.518 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:46:55.533 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:47:05.552 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:47:15.567 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:47:25.582 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:47:35.601 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:47:45.620 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:47:55.637 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:48:05.658 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:48:15.696 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:48:25.714 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:48:35.732 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:48:45.748 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:48:55.766 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:49:05.784 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:49:15.798 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:49:25.807 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:49:35.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:49:45.834 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:49:55.851 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:50:05.871 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:50:15.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:50:25.895 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:50:35.914 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:50:45.926 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:50:55.945 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:51:05.959 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:51:15.975 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:51:25.985 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:51:35.996 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:51:46.015 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:51:56.031 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:52:06.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:52:16.067 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:52:26.084 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:52:36.104 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:52:46.123 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:52:56.135 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:53:06.155 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:53:16.376 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:53:26.395 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:53:36.414 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:53:46.433 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:53:56.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:54:06.467 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:54:16.477 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:54:26.497 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:54:36.510 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:54:46.521 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:54:56.538 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:55:06.554 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:55:16.572 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:55:26.591 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:55:36.607 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:55:46.625 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:55:56.644 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:56:06.662 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:56:16.678 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:56:26.697 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:56:36.708 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:56:46.719 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)... 2020-11-16T21:56:56.734 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:57:06.753 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:57:16.772 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:57:26.791 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:57:36.809 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:57:46.828 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:57:56.846 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:58:06.865 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:58:16.884 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:58:26.903 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:58:36.925 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:58:46.937 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:58:56.956 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T21:59:06.974 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:59:16.993 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:59:27.010 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:59:37.019 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:59:47.037 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T21:59:57.056 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:00:07.071 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:00:17.091 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:00:27.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:00:37.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:00:47.148 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:00:57.166 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:07.185 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:17.202 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:27.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:37.235 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:47.297 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:01:57.316 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:07.333 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:17.404 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:27.423 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:37.437 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:47.455 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:02:57.473 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:07.491 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:17.509 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:27.527 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:37.543 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:47.560 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:03:57.576 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:07.595 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:17.610 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:27.630 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:37.647 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:47.664 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:04:57.682 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:07.698 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:17.715 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:27.725 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:37.745 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:47.764 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:05:57.776 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:07.787 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:17.805 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:27.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:37.841 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:47.858 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:06:57.874 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:07.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:17.904 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:27.918 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:37.929 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:47.946 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:07:57.963 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:07.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:17.999 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:28.016 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:38.028 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:48.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:08:58.062 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:08.071 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:18.091 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:28.107 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:38.126 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:48.142 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:09:58.154 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:10:08.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)... 2020-11-16T22:10:18.183 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:10:28.202 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:10:38.220 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:10:48.237 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:10:58.253 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:08.270 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:18.295 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:28.323 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:38.339 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:48.357 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:11:58.373 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:08.388 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:18.405 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:28.424 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:38.441 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:48.457 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:12:58.470 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:08.479 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:18.494 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:28.503 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:38.518 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:48.533 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:13:58.548 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:08.567 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:18.578 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:28.595 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:38.611 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:48.631 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:14:58.650 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:08.666 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:18.683 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:28.698 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:38.714 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:48.726 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:15:58.745 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:08.756 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:18.774 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:28.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:38.803 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:48.820 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:16:58.835 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:08.853 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:18.871 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:28.891 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:38.909 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:48.927 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:17:58.937 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:08.955 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:18.974 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:28.990 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:39.009 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:49.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:18:59.035 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:19:09.056 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:19:19.074 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:19:29.093 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:19:39.114 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:20:15.059 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:20:38.746 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:20:48.758 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:20:58.774 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:21:08.792 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:21:18.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:21:28.827 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:21:38.843 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:21:48.862 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:21:58.875 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:22:08.892 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:22:18.905 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:22:28.921 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:22:38.938 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:22:48.950 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:22:58.969 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:23:08.987 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:23:19.000 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:23:29.016 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:23:39.033 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:23:49.042 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:23:59.059 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:24:09.073 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:24:19.091 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:24:29.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:24:39.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:24:49.137 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:24:59.154 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:09.170 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:19.186 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:29.202 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:39.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:49.233 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:25:59.248 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:09.261 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:19.279 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:29.294 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:39.303 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:49.320 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:26:59.338 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:09.355 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:19.371 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:29.386 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:39.402 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:49.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:27:59.429 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:09.446 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:19.463 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:29.479 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:39.490 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:49.499 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:28:59.516 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:09.535 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:19.551 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:29.566 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:39.581 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:49.605 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:29:59.616 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:09.633 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:19.651 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:29.666 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:39.685 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:49.701 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:30:59.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:09.736 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:19.749 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:29.766 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:39.783 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:49.800 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:31:59.817 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:09.832 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:19.848 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:29.864 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:39.876 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:49.892 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:32:59.909 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:09.925 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:19.941 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:29.949 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:39.965 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:33:49.984 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:00.000 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:10.019 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:20.035 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:30.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:40.067 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:34:50.085 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:35:00.101 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:35:10.117 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:35:20.134 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:35:30.151 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:35:40.168 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:35:50.186 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 3, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-11-16_20:59:43-multimds-wip-pdonnell-testing-20201116.174825-distro-basic-smithi/5631931'} 2020-11-16T22:35:50.194 ERROR:teuthology.lock.ops:Insufficient nodes available to lock 3 smithi nodes. 2020-11-16T22:35:50.195 ERROR:teuthology.lock.ops:{"message": "only 0 nodes available"} 2020-11-16T22:35:50.195 INFO:teuthology.task.internal.lock_machines:0 smithi machines locked this try, 0/3 locked so far 2020-11-16T22:35:50.195 INFO:teuthology.task.internal.lock_machines:0 machines locked (0 new); need 3 more 2020-11-16T22:35:50.196 WARNING:teuthology.task.internal.lock_machines:Could not lock enough machines, waiting... 2020-11-16T22:36:00.210 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:36:10.227 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:36:20.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:36:30.259 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:36:40.273 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:36:50.288 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:37:00.298 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:37:10.313 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:37:20.329 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:37:30.344 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:37:40.361 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:38:13.019 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:38:23.035 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:38:33.051 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:38:43.065 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:38:53.074 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:39:03.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:39:13.106 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:39:23.117 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:39:33.133 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:39:43.151 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:39:53.168 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:40:03.186 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:40:13.203 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:40:23.216 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:40:33.233 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:40:43.249 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:40:57.340 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:41:09.393 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:41:24.507 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:41:39.215 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:41:51.272 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:42:06.391 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:42:21.511 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T22:42:54.392 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)... 2020-11-16T23:00:09.304 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...