2020-03-09T00:52:42.637 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-09T00:52:42.638 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.pQItm6.tmp -- --verbose --lock --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-partial-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml supported-random-distro$/{rhel_8.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/redirect.yaml} --name sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi --block --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838273 2020-03-09T00:52:42.678 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-09T00:52:42.748 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi/4838273 branch: wip-sage4-testing-2020-03-08-0938 description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-partial-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp-zstd.yaml rados.yaml supported-random-distro$/{rhel_8.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/redirect.yaml} email: null first_in_suite: false job_id: '4838273' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2020-03-08_17:10:32-rados-wip-sage4-testing-2020-03-08-0938-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: rhel os_version: '8.1' overrides: admin_socket: branch: wip-sage4-testing-2020-03-08-0938 ceph: conf: global: mon client directed command retry: 5 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd_object_clean_region_max_num_intervals: 1000 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on degraded: true osd backoff on peering: true osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd delete sleep: 1 osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-sage4-testing-2020-03-08-0938 sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b owner: scheduled_sage@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b suite: rados suite_branch: wip-sage4-testing-2020-03-08-0938 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 low_tier_pool: low_tier objects: 500 op_weights: copy_from: 50 delete: 50 read: 100 write: 100 ops: 4000 set_redirect: true teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.490 2020-03-09T00:52:42.749 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa; will attempt to use it 2020-03-09T00:52:42.750 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage4-testing-2020-03-08-0938/qa/tasks 2020-03-09T00:52:42.750 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-09T00:52:42.752 INFO:teuthology.task.internal:Checking packages... 2020-03-09T00:52:42.779 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '9eb9490ed0531fd33292066429bf9db55b9c1d3b' 2020-03-09T00:52:42.780 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-09T00:52:42.780 INFO:teuthology.packaging:ref: None 2020-03-09T00:52:42.780 INFO:teuthology.packaging:tag: None 2020-03-09T00:52:42.780 INFO:teuthology.packaging:branch: wip-sage4-testing-2020-03-08-0938 2020-03-09T00:52:42.780 INFO:teuthology.packaging:sha1: 9eb9490ed0531fd33292066429bf9db55b9c1d3b 2020-03-09T00:52:42.780 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-sage4-testing-2020-03-08-0938 2020-03-09T00:52:42.943 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1904.g9eb9490 2020-03-09T00:52:42.943 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-09T00:52:42.951 INFO:teuthology.task.internal:no buildpackages task found 2020-03-09T00:52:42.951 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-09T00:52:42.978 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-09T00:52:42.978 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-09T00:52:43.022 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T00:52:53.036 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T00:53:03.054 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T00:53:13.072 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T00:53:23.084 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T00:53:33.102 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T00:53:43.117 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T00:53:53.135 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T00:54:03.153 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T00:54:13.171 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:54:23.186 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:54:33.204 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:54:43.222 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:54:53.235 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:55:03.256 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:55:13.270 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:55:23.285 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:55:33.302 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:55:43.314 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:55:53.332 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:56:03.343 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:56:13.361 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:56:23.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:56:33.396 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:56:43.406 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:56:53.425 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:57:03.443 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:57:13.461 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:57:23.479 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:57:33.491 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:57:43.501 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:57:53.516 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:58:03.533 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:58:13.548 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:58:23.557 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:58:33.576 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:58:43.590 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:58:53.606 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:59:03.623 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:59:13.641 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:59:23.653 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:59:33.670 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:59:43.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T00:59:53.703 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:00:03.722 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:00:13.740 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:00:23.756 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:00:33.771 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:00:43.787 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:00:53.802 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:01:03.825 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:01:13.849 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:01:23.870 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:01:33.884 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:01:43.898 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:01:53.910 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:02:03.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:02:13.942 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:02:23.960 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:02:33.975 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:02:43.990 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:02:54.000 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:03:04.015 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:03:14.033 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:03:24.046 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:03:34.063 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:03:44.072 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:03:54.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:04:04.100 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:04:14.117 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:04:24.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:04:34.142 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:04:44.159 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:04:54.176 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:05:04.194 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:05:14.212 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:05:24.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:05:34.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:05:44.256 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:05:54.275 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:06:04.292 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:06:14.310 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:06:24.335 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:06:34.344 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:06:44.359 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:06:54.377 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:07:04.387 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:07:14.402 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:07:24.415 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:07:34.431 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:07:44.446 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:07:54.457 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:08:04.474 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:08:14.490 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:08:24.508 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:08:34.522 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:08:44.542 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:08:54.559 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:09:04.577 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:09:14.591 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:09:24.611 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:09:34.623 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:09:44.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:09:54.656 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:10:04.666 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:10:14.682 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:10:24.700 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:10:34.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:10:44.736 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:10:54.754 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:11:04.772 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:11:14.788 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:11:24.806 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:11:34.822 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:11:44.835 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:11:54.851 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:12:04.865 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:12:14.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:12:24.896 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:12:34.910 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:12:44.927 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:12:54.945 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:13:04.959 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:13:14.970 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:13:24.986 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:13:34.996 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:13:45.012 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:13:55.030 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:14:05.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:14:15.062 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:14:25.079 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:14:35.096 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:14:45.110 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:14:55.129 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:15:05.147 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:15:15.165 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:15:25.183 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:15:35.202 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:15:45.219 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:15:55.231 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:16:05.246 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:16:15.264 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:16:25.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:16:35.301 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:16:45.318 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:16:55.335 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:17:05.359 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:17:15.380 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:17:25.398 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:17:35.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:17:45.437 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:17:55.459 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:18:05.480 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:18:15.497 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:18:25.514 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:18:35.528 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:18:45.546 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:18:55.565 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:19:05.583 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:19:15.601 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:19:25.619 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-03-09T01:19:35.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:19:45.654 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:19:55.670 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:20:05.690 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:20:15.708 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:20:25.727 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:20:35.745 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:20:45.757 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:20:55.774 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:21:05.792 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:21:15.810 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:21:25.824 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:21:35.839 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:21:45.856 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:21:55.866 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:22:05.885 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:22:15.896 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:22:25.914 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:22:35.934 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:22:45.953 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:22:55.964 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:23:05.978 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:23:15.995 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:23:26.014 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:23:36.032 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:23:46.050 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:23:56.069 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:24:06.087 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:24:16.100 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:24:26.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:24:36.128 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:24:46.146 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:24:56.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:25:06.182 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:25:16.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:25:26.214 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:25:36.231 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:25:46.246 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:25:56.261 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:26:06.280 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:26:16.301 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:26:26.317 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:26:36.333 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:26:46.351 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:26:56.368 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:27:06.383 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:27:16.397 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:27:26.415 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:27:36.430 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:27:46.452 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:27:56.464 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:28:06.484 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:28:16.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:28:26.515 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:28:36.531 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:28:46.548 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:28:56.567 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:29:06.582 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:29:16.598 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:29:26.613 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:29:36.630 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:29:46.647 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:29:56.662 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:30:06.678 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:30:16.695 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:30:26.711 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2020-03-09T01:30:36.732 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)...