2020-03-27T18:07:35.663 INFO:root:teuthology version: 1.0.0-b3a6b1a 2020-03-27T18:07:35.664 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.MW19DA.tmp -- --verbose --lock --description rados:thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/mimic.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_7.6.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/careful.yaml thrashosds-health.yaml workloads/rbd_cls.yaml} --name sage-2020-03-27_17:51:50-rados:thrash-old-clients-revert-dec-distro-basic-smithi --block --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2020-03-27_17:51:50-rados:thrash-old-clients-revert-dec-distro-basic-smithi/4895634 2020-03-27T18:07:35.701 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T18:07:35.760 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2020-03-27_17:51:50-rados:thrash-old-clients-revert-dec-distro-basic-smithi/4895634 branch: revert-dec description: rados:thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/mimic.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_7.6.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/careful.yaml thrashosds-health.yaml workloads/rbd_cls.yaml} email: null first_in_suite: false job_id: '4895634' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'rbd object class functional tests ' name: sage-2020-03-27_17:51:50-rados:thrash-old-clients-revert-dec-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: '7.6' overrides: admin_socket: branch: revert-dec ceph: conf: global: mon client directed command retry: 5 mon warn on msgr2 not enabled: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd backoff on peering: true osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request mon_bind_msgr2: false sha1: df873c1654814c23dcb68f897a012936832bfab9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: df873c1654814c23dcb68f897a012936832bfab9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: revert-dec sha1: df873c1654814c23dcb68f897a012936832bfab9 owner: scheduled_sage@teuthology priority: 50 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 sha1: df873c1654814c23dcb68f897a012936832bfab9 suite: rados:thrash-old-clients suite_branch: revert-dec suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_revert-dec/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: df873c1654814c23dcb68f897a012936832bfab9 tasks: - install: branch: mimic exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - cephadm - ceph-immutable-object-cache - ceph-base - python3-rados - python3-rgw - python3-rbd - python3-cephfs - librados-devel extra_packages: - librados2 - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: null - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.2: - ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2735 2020-03-27T18:07:35.760 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_revert-dec/qa; will attempt to use it 2020-03-27T18:07:35.761 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_revert-dec/qa/tasks 2020-03-27T18:07:35.761 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-27T18:07:35.764 INFO:teuthology.task.internal:Checking packages... 2020-03-27T18:07:35.777 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'df873c1654814c23dcb68f897a012936832bfab9' 2020-03-27T18:07:35.777 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-27T18:07:35.777 INFO:teuthology.packaging:ref: None 2020-03-27T18:07:35.777 INFO:teuthology.packaging:tag: None 2020-03-27T18:07:35.777 INFO:teuthology.packaging:branch: revert-dec 2020-03-27T18:07:35.777 INFO:teuthology.packaging:sha1: df873c1654814c23dcb68f897a012936832bfab9 2020-03-27T18:07:35.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=revert-dec 2020-03-27T18:07:36.015 INFO:teuthology.task.internal:Found packages for ceph version 15.2.0-377.gdf873c1 2020-03-27T18:07:36.015 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-27T18:07:36.030 INFO:teuthology.task.internal:no buildpackages task found 2020-03-27T18:07:36.030 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-27T18:07:36.071 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-27T18:07:36.072 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-27T18:07:36.126 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:07:46.141 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:07:56.157 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:08:06.173 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:08:16.188 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:08:26.209 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:08:36.224 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:08:46.241 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:08:56.254 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:09:06.269 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:09:16.286 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:09:26.304 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:09:36.526 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:09:46.541 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:09:56.557 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:10:06.574 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 0)... 2020-03-27T18:10:16.589 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:10:26.605 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:10:36.623 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:10:46.636 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:10:56.681 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:11:06.697 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:11:16.714 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:11:26.729 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:11:36.746 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:11:46.761 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:11:56.774 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:12:06.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:12:16.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:12:26.825 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:12:36.841 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:12:46.863 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:12:56.877 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:13:06.893 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:13:16.909 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:13:26.927 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:13:36.936 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:13:46.952 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:13:56.985 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:14:07.001 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-03-27T18:14:17.020 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:14:27.041 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:14:37.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:14:47.081 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:14:57.097 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:15:07.115 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:15:17.129 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:15:27.146 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:15:37.161 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:15:47.177 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:15:57.193 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:16:07.209 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-03-27T18:16:17.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:16:27.242 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:16:37.258 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:16:47.275 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:16:57.293 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:17:07.308 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:17:17.324 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:17:27.341 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:17:37.357 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:17:47.372 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:17:57.391 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:18:07.408 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:18:17.428 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:18:27.447 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:18:37.464 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:18:47.481 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:18:57.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:19:07.513 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:19:17.537 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:19:27.554 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:19:37.571 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:19:47.587 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:19:57.601 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:20:07.621 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:20:17.651 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:20:27.671 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:20:37.684 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:20:47.697 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:20:57.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:21:07.737 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:21:17.754 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:21:27.771 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:21:37.787 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:21:47.805 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:21:57.825 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:22:07.847 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:22:17.877 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:22:27.903 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:22:37.937 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:22:47.954 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:22:57.968 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:23:07.985 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:23:18.003 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:23:28.020 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:23:38.039 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:23:48.054 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:23:58.068 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:24:08.101 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:24:18.119 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:24:28.137 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:24:38.153 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:24:48.174 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:24:58.190 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:25:08.206 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:25:18.222 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:25:28.237 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:25:38.253 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:25:48.268 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:25:58.285 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:26:08.302 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:26:18.320 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:26:28.337 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:26:38.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:26:48.380 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:26:58.396 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:27:08.413 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:27:18.430 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:27:28.453 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:27:38.469 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:27:48.484 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:27:58.510 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:28:08.527 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:28:18.541 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:28:28.557 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:28:38.572 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:28:48.588 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:28:58.606 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:29:08.625 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:29:18.640 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:29:28.656 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:29:38.677 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:29:48.694 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:29:58.711 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:30:08.729 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:30:18.745 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:30:28.763 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-03-27T18:30:38.776 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:30:48.795 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:30:58.809 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:31:08.835 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:31:18.854 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:31:28.867 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:31:38.884 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:31:48.913 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:31:58.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:32:08.945 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:32:18.961 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:32:28.996 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:32:39.014 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:32:49.030 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:32:59.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:33:09.068 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:33:19.085 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:33:29.101 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:33:39.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:33:49.139 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:33:59.160 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:34:09.177 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:34:19.194 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:34:29.226 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:34:39.267 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:34:49.281 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:34:59.312 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:35:09.330 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:35:19.348 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:35:29.372 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:35:39.405 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:35:49.421 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:35:59.458 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:36:09.472 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:36:19.490 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:36:29.504 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:36:39.521 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:36:49.539 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:36:59.552 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:37:09.565 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-03-27T18:37:19.584 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:37:29.602 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:37:39.614 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:37:49.636 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:37:59.661 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:38:09.682 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:38:19.711 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:38:29.725 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:38:39.741 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:38:49.772 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:38:59.794 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:39:09.811 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-03-27T18:39:19.825 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)...