2020-01-27T18:36:08.258 INFO:root:teuthology version: 1.0.0-7063d1e
2020-01-27T18:36:08.258 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.pHl7ar.tmp -- --verbose --lock --description upgrade:mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml rgw_swift.yaml} objectstore/filestore-xfs.yaml supported-all-distro/rhel_7.yaml} --name dzafman-2020-01-25_01:21:34-upgrade:mimic-x-master-distro-basic-smithi --block --owner scheduled_dzafman@ubuntu-1804 --archive /home/teuthworker/archive/dzafman-2020-01-25_01:21:34-upgrade:mimic-x-master-distro-basic-smithi/4711857
2020-01-27T18:36:08.337 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-01-27T18:36:08.413 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/dzafman-2020-01-25_01:21:34-upgrade:mimic-x-master-distro-basic-smithi/4711857
  branch: master
  description: upgrade:mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml
    1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml
    rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml
    test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-msgr2.yaml 4-nautilus.yaml
    5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml
    rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml
    rgw_swift.yaml} objectstore/filestore-xfs.yaml supported-all-distro/rhel_7.yaml}
  email: dzafman@redhat.com
  first_in_suite: false
  job_id: '4711857'
  kernel:
    kdb: true
    sha1: distro
  last_in_suite: false
  machine_type: smithi
  meta:
  - desc: 'Run ceph on two nodes,
  
      with a separate client 0,1,2 third node.
  
      Use xfs beneath the osds.
  
      CephFS tests running on client 2,3
  
      #Note-To enable RHEL runs on ovh nodes, add the following to overrides
  
      #ansible.cephlab:
  
      # skip_tags: entitlements,packages,repos
  
      '
  - desc: 'install ceph/mimic latest
  
      run workload and upgrade-sequence in parallel
  
      upgrade the client node
  
      '
  - desc: 'run a cephfs stress test
  
      mount ceph-fuse on client.2 before running workunit
  
      '
  - desc: 'run run randomized correctness test for rados operations
  
      on an erasure-coded pool
  
      '
  - desc: 'object class functional tests
  
      '
  - desc: 'generate read/write load with rados objects ranging from 1MB to 25MB
  
      '
  - desc: 'rgw ragweed prepare
  
      '
  - desc: 'librbd C and C++ api tests
  
      '
  - desc: 'librbd python api tests
  
      '
  - desc: 'upgrade the ceph cluster
  
      '
  - desc: 'run a cephfs stress test
  
      mount ceph-fuse on client.3 before running workunit
  
      '
  - desc: 'randomized correctness test for rados operations on a replicated pool with
      snapshots
  
      '
  - desc: 'generate read/write load with rados objects ranging from 1 byte to 1MB
  
      '
  - desc: 'librados C and C++ api tests
  
      '
  - desc: 'rbd object class functional tests
  
      '
  - desc: 'run basic import/export cli tests for rbd
  
      '
  - desc: 'ragweed check for rgw
  
      '
  - desc: 'swift api tests for rgw
  
      '
  name: dzafman-2020-01-25_01:21:34-upgrade:mimic-x-master-distro-basic-smithi
  nuke-on-error: true
  openstack:
  - volumes:
      count: 3
      size: 30
  os_type: rhel
  os_version: '7.7'
  overrides:
    admin_socket:
      branch: master
    ceph:
      conf:
        global:
          enable experimental unrecoverable data corrupting features: '*'
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
          mon warn on osd down out interval zero: false
        osd:
          debug filestore: 20
          debug journal: 20
          debug ms: 1
          debug osd: 25
          osd objectstore: filestore
          osd sloppy crc: true
          osd_class_default_list: '*'
          osd_class_load_list: '*'
      fs: xfs
      log-whitelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - scrub mismatch
      - ScrubResult
      - wrongly marked
      - \(POOL_APP_NOT_ENABLED\)
      - \(SLOW_OPS\)
      - overall HEALTH_
      - \(MON_MSGR2_NOT_ENABLED\)
      - reached quota
      - \(REQUEST_SLOW\)
      mon_bind_addrvec: false
      mon_bind_msgr2: false
      sha1: 9c19d0edb97c85432761f769828070f563587117
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon:
          osd default pool size: 2
        osd:
          osd objectstore: filestore
          osd sloppy crc: true
      filestore: true
      fs: xfs
    install:
      ceph:
        sha1: 9c19d0edb97c85432761f769828070f563587117
    rgw:
      frontend: civetweb
    workunit:
      branch: master
      sha1: 35b0b52153206fd6178afe584a2cd9c0ef2bcc05
  owner: scheduled_dzafman@ubuntu-1804
  priority: 50
  repo: git://git.ceph.com/ceph-ci.git
  rgw-final-workload:
    full_sequential:
    - ragweed:
        client.1:
          default-branch: ceph-master
          rgw_server: client.1
          stages: check
    - print: '**** done ragweed check 4-final-workload'
    - swift:
        client.1:
          force-branch: ceph-master
          rgw_server: client.1
    - print: '**** done swift 4-final-workload'
  roles:
  - - mon.a
    - mgr.x
    - mds.a
    - osd.0
    - osd.1
  - - mon.b
    - mgr.y
    - osd.2
    - osd.3
  - - mon.c
    - mds.b
    - osd.4
    - osd.5
  - - client.0
    - client.1
    - client.2
    - client.3
  sha1: 9c19d0edb97c85432761f769828070f563587117
  suite: upgrade:mimic-x
  suite_branch: master
  suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_master/qa
  suite_relpath: qa
  suite_repo: git://git.ceph.com/ceph-ci.git
  suite_sha1: 35b0b52153206fd6178afe584a2cd9c0ef2bcc05
  tasks:
  - install:
      branch: mimic
      exclude_packages:
      - librados3
      - ceph-mgr-dashboard
      - ceph-mgr-diskprediction-local
      - ceph-mgr-diskprediction-cloud
      - ceph-mgr-rook
      - ceph-mgr-ssh
      - ceph-daemon
      extra_packages:
      - librados2
  - print: '**** done installing mimic'
  - ceph:
      conf:
        global:
          bluestore warn on no per pool omap: false
          bluestore_warn_on_legacy_statfs: false
          mon warn on pool no app: false
      log-whitelist:
      - overall HEALTH_
      - \(FS_
      - \(MDS_
      - \(OSD_
      - \(MON_DOWN\)
      - \(CACHE_POOL_
      - \(POOL_
      - \(MGR_DOWN\)
      - \(PG_
      - \(SMALLER_PGP_NUM\)
      - Monitor daemon marked osd
      - Behind on trimming
      - Manager daemon
  - exec:
      osd.0:
      - ceph osd require-osd-release mimic
      - ceph osd set-require-min-compat-client mimic
  - print: '**** done ceph'
  - install.upgrade:
      mon.a: null
      mon.b: null
      mon.c: null
  - print: '**** done install.upgrade non-client hosts'
  - rgw:
    - client.1
  - print: '**** done => started rgw client.1'
  - parallel:
    - workload
    - upgrade-sequence
  - print: '**** done parallel'
  - install.upgrade:
      client.0: null
  - print: '**** done install.upgrade on client.0'
  - exec:
      mon.a:
      - ceph mon enable-msgr2
      - ceph config rm global mon_warn_on_msgr2_not_enabled
  - exec:
      osd.0:
      - ceph osd require-osd-release nautilus
      - ceph osd set-require-min-compat-client nautilus
      - for p in `ceph osd pool ls`; do ceph osd pool set $p pg_autoscale_mode off;
        done
  - ceph.healthy: null
  - sequential:
    - ceph-fuse: null
    - print: '**** done ceph-fuse 5-final-workload'
    - workunit:
        clients:
          client.3:
          - suites/blogbench.sh
    - print: '**** done suites/blogbench.sh 5-final-workload'
  - rados:
      clients:
      - client.1
      objects: 50
      op_weights:
        delete: 50
        read: 100
        rollback: 50
        snap_create: 50
        snap_remove: 50
        write: 100
      ops: 4000
      write_append_excl: false
  - print: '**** done rados 4-final-workload'
  - workunit:
      clients:
        client.1:
        - rados/load-gen-mix.sh
  - print: '**** done rados/load-gen-mix.sh 4-final-workload'
  - mon_thrash:
      revive_delay: 20
      thrash_delay: 1
  - print: '**** done mon_thrash 4-final-workload'
  - workunit:
      clients:
        client.1:
        - rados/test.sh
  - print: '**** done rados/test.sh 4-final-workload'
  - workunit:
      clients:
        client.1:
        - cls/test_cls_rbd.sh
  - print: '**** done cls/test_cls_rbd.sh 4-final-workload'
  - workunit:
      clients:
        client.1:
        - rbd/import_export.sh
      env:
        RBD_CREATE_ARGS: --new-format
  - print: '**** done rbd/import_export.sh 4-final-workload'
  - sequential:
    - rgw-final-workload
    - print: '**** done rgw 4-final-workload'
  teuthology_branch: master
  tube: smithi
  upgrade-sequence:
    sequential:
    - ceph.restart:
        daemons:
        - mon.a
        - mon.b
        - mon.c
        - mgr.x
        - mgr.y
        mon-health-to-clog: false
        wait-for-healthy: false
    - exec:
        mon.a:
        - ceph config set global mon_warn_on_msgr2_not_enabled false
    - ceph.healthy: null
    - ceph.restart:
        daemons:
        - osd.0
        - osd.1
        - osd.2
        - osd.3
        - osd.4
        - osd.5
        wait-for-healthy: false
        wait-for-osds-up: true
    - ceph.restart:
        daemons:
        - mds.a
        - mds.b
        - rgw.*
        wait-for-healthy: false
        wait-for-osds-up: true
    - print: '**** done ceph.restart all'
  verbose: false
  worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2505
  workload:
    full_sequential:
    - sequential:
      - ceph-fuse: null
      - print: '**** done ceph-fuse 2-workload'
      - workunit:
          clients:
            client.2:
            - suites/blogbench.sh
      - print: '**** done suites/blogbench.sh 2-workload'
    - rados:
        clients:
        - client.0
        ec_pool: true
        objects: 50
        op_weights:
          append: 100
          copy_from: 50
          delete: 50
          read: 100
          rmattr: 25
          rollback: 50
          setattr: 25
          snap_create: 50
          snap_remove: 50
          write: 0
        ops: 4000
        write_append_excl: false
    - print: '**** done rados ec task'
    - workunit:
        branch: mimic
        clients:
          client.0:
          - cls
    - print: '**** done cls 2-workload'
    - workunit:
        branch: mimic
        clients:
          client.0:
          - rados/load-gen-big.sh
    - print: '**** done rados/load-gen-big.sh 2-workload'
    - sequential:
      - ragweed:
          client.1:
            default-branch: ceph-master
            rgw_server: client.1
            stages: prepare
      - print: '**** done rgw ragweed prepare 2-workload'
    - workunit:
        branch: mimic
        clients:
          client.0:
          - rbd/test_librbd.sh
    - print: '**** done rbd/test_librbd.sh 2-workload'
    - workunit:
        branch: mimic
        clients:
          client.0:
          - rbd/test_librbd_python.sh
    - print: '**** done rbd/test_librbd_python.sh 2-workload'
2020-01-27T18:36:08.413 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_master/qa; will attempt to use it
2020-01-27T18:36:08.414 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_master/qa/tasks
2020-01-27T18:36:08.414 INFO:teuthology.run_tasks:Running task internal.check_packages...
2020-01-27T18:36:08.415 INFO:teuthology.task.internal:Checking packages...
2020-01-27T18:36:08.427 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '9c19d0edb97c85432761f769828070f563587117'
2020-01-27T18:36:08.427 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2020-01-27T18:36:08.427 INFO:teuthology.packaging:ref: None
2020-01-27T18:36:08.428 INFO:teuthology.packaging:tag: None
2020-01-27T18:36:08.428 INFO:teuthology.packaging:branch: master
2020-01-27T18:36:08.428 INFO:teuthology.packaging:sha1: 9c19d0edb97c85432761f769828070f563587117
2020-01-27T18:36:08.428 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=master
2020-01-27T18:36:09.699 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-9849.ge33f610
2020-01-27T18:36:09.700 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2020-01-27T18:36:09.708 INFO:teuthology.task.internal:no buildpackages task found
2020-01-27T18:36:09.708 INFO:teuthology.run_tasks:Running task internal.lock_machines...
2020-01-27T18:36:09.766 INFO:teuthology.task.internal.lock_machines:Locking machines...
2020-01-27T18:36:09.766 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-01-27T18:36:09.818 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:36:19.836 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:36:29.855 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:36:39.866 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:36:49.880 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:36:59.899 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:37:09.917 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:37:19.934 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:37:29.948 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:37:39.966 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:37:49.984 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:38:00.002 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:38:10.012 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:38:20.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)...
2020-01-27T18:38:30.042 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)...
2020-01-27T18:38:40.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)...
2020-01-27T18:38:50.080 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)...
2020-01-27T18:39:00.099 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:39:10.111 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:39:20.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:39:30.144 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:39:40.160 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:39:50.177 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:40:00.196 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:40:10.214 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:40:20.231 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:40:30.250 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:40:40.270 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:40:50.288 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:41:00.307 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:41:10.322 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:41:20.339 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:41:30.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:41:40.377 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:41:50.395 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:42:00.412 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:42:10.424 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:42:20.440 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:42:30.460 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:42:40.479 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:42:50.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:43:00.520 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:43:10.539 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:43:20.555 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:43:30.571 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:43:40.590 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:43:50.607 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:44:00.623 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:44:10.639 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:44:20.654 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:44:30.664 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:44:40.675 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:44:50.694 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:45:00.736 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:45:10.755 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:45:20.770 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:45:30.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:45:40.809 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:45:50.826 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:46:00.845 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:46:10.863 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:46:20.887 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:46:30.903 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:46:40.919 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:46:50.934 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:47:00.952 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:47:10.971 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:47:20.989 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:47:31.001 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:47:41.020 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:47:51.035 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:48:01.052 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:48:11.070 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:48:21.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:48:31.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:48:41.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:48:51.145 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:49:01.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:49:11.182 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:49:21.199 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:49:31.210 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:49:41.226 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:49:51.244 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:50:01.262 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:50:11.280 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:50:21.294 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:50:31.315 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:50:41.331 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:50:51.347 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:51:01.363 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:51:11.379 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:51:21.396 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:51:31.414 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:51:41.430 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:51:51.442 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:52:01.458 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:52:11.475 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:52:21.492 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:52:31.508 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:52:41.526 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:52:51.542 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:53:01.560 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:53:11.578 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:53:21.602 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:53:31.621 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:53:41.640 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:53:51.658 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:54:01.676 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:54:11.694 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T18:54:21.710 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:54:31.727 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:54:41.743 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:54:51.753 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T18:55:01.767 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:55:11.784 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:55:21.794 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:55:31.812 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:55:41.830 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:55:51.845 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:56:01.863 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:56:11.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:56:21.901 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:56:31.915 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:56:41.929 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:56:51.948 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:57:01.963 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:57:13.093 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:57:23.110 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:57:33.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:57:43.143 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:57:53.160 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:58:03.178 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:58:13.190 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:58:23.207 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:58:33.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:58:43.244 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:58:53.260 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:59:03.277 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:59:13.295 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:59:23.310 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:59:33.326 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T18:59:43.350 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)...
2020-01-27T18:59:53.365 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:00:03.383 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:00:13.398 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:00:23.412 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:00:33.427 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:00:43.445 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:00:53.462 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:01:03.480 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:01:13.500 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:01:23.510 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:01:33.528 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:01:43.538 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:01:53.562 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:02:03.573 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:02:13.590 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:02:23.608 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:02:33.619 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:02:43.634 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:02:53.649 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:03:03.667 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:03:13.683 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:03:23.698 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:03:33.715 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:03:43.731 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)...
2020-01-27T19:03:53.747 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:04:03.763 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:04:13.780 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:04:23.796 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:04:33.815 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:04:43.830 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:04:53.839 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:05:03.857 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:05:13.867 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:05:23.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:05:33.904 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:05:43.920 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:05:53.936 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:06:03.951 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:06:13.967 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:06:23.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:06:33.994 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:06:44.009 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:06:54.027 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:07:04.046 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:07:14.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:07:24.078 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:07:34.094 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:07:44.110 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:07:54.125 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:08:04.143 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:08:14.160 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:08:24.177 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:08:34.190 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:08:44.207 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:08:54.221 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:09:04.233 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:09:14.249 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:09:24.260 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:09:34.279 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:09:44.294 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:09:54.302 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:10:04.318 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:10:14.335 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:10:24.348 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:10:34.362 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:10:44.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:10:54.390 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:11:04.412 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:11:14.428 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:11:24.447 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:11:34.464 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:11:44.478 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:11:54.486 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:12:04.504 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:12:14.536 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:12:24.558 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:12:34.575 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:12:44.592 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:12:54.600 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:13:04.612 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:13:14.625 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:13:24.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:13:34.648 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:13:44.664 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:13:54.677 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:14:04.695 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:14:14.714 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:14:24.733 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:14:34.750 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:14:44.768 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:14:54.782 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:15:04.800 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:15:14.822 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:15:24.838 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:15:34.850 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:15:44.869 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:15:54.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T19:16:04.893 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:16:14.910 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:16:24.928 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:16:34.944 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:16:44.962 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:16:54.972 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:17:04.990 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:17:15.009 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:17:25.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:17:35.044 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:17:45.059 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:17:55.075 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:18:05.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:18:15.100 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:18:25.117 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:18:35.135 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:18:45.150 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T19:18:55.166 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)...
2020-01-27T19:19:05.183 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)...
2020-01-27T19:19:15.197 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:19:25.213 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:19:35.231 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:19:45.242 DEBUG:teuthology.lock.ops:lock_many request: {'count': 4, 'locked_by': 'scheduled_dzafman@ubuntu-1804', 'description': '/home/teuthworker/archive/dzafman-2020-01-25_01:21:34-upgrade:mimic-x-master-distro-basic-smithi/4711857', 'machine_type': 'smithi'}
2020-01-27T19:19:45.266 ERROR:teuthology.lock.ops:Insufficient nodes available to lock 4 smithi nodes.
2020-01-27T19:19:45.266 ERROR:teuthology.lock.ops:{"message": "only 3 nodes available"}
2020-01-27T19:19:45.266 INFO:teuthology.task.internal.lock_machines:0 smithi machines locked this try, 0/4 locked so far
2020-01-27T19:19:45.266 INFO:teuthology.task.internal.lock_machines:0 machines locked (0 new); need 4 more
2020-01-27T19:19:45.266 WARNING:teuthology.task.internal.lock_machines:Could not lock enough machines, waiting...
2020-01-27T19:19:55.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:20:05.299 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:20:15.314 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:20:25.325 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:20:35.342 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:20:45.360 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:20:55.374 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:21:05.392 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:21:15.406 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:21:25.425 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:21:35.445 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:21:45.462 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:21:55.481 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:22:05.499 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)...
2020-01-27T19:22:15.518 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:22:25.534 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:22:35.551 DEBUG:teuthology.lock.ops:lock_many request: {'count': 4, 'locked_by': 'scheduled_dzafman@ubuntu-1804', 'description': '/home/teuthworker/archive/dzafman-2020-01-25_01:21:34-upgrade:mimic-x-master-distro-basic-smithi/4711857', 'machine_type': 'smithi'}
2020-01-27T19:22:35.573 ERROR:teuthology.lock.ops:Insufficient nodes available to lock 4 smithi nodes.
2020-01-27T19:22:35.573 ERROR:teuthology.lock.ops:{"message": "only 2 nodes available"}
2020-01-27T19:22:35.573 INFO:teuthology.task.internal.lock_machines:0 smithi machines locked this try, 0/4 locked so far
2020-01-27T19:22:35.574 INFO:teuthology.task.internal.lock_machines:0 machines locked (0 new); need 4 more
2020-01-27T19:22:35.574 WARNING:teuthology.task.internal.lock_machines:Could not lock enough machines, waiting...
2020-01-27T19:22:45.592 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:22:55.607 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:23:05.626 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:23:15.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:23:25.651 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:23:35.668 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:23:45.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:23:55.704 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:24:05.724 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:24:15.737 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:24:25.752 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:24:35.768 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:24:45.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:24:55.807 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:25:05.822 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:25:15.834 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:25:25.847 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:25:35.865 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:25:45.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:25:55.899 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:26:05.914 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:26:15.932 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:26:25.942 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:26:35.958 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:26:45.976 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:26:55.994 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:27:06.012 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:27:16.026 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:27:26.044 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:27:36.062 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:27:46.078 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:27:56.095 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:28:06.111 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:28:16.125 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...
2020-01-27T19:28:26.146 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)...