2020-08-29T11:01:22.277 INFO:root:teuthology version: 1.0.0-d7ff7d9
2020-08-29T11:01:22.279 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name pdonnell-2020-08-29_06:13:10-multimds-wip-pdonnell-testing-20200829.021135-distro-basic-smithi --lock --description multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_exports} --archive /home/teuthworker/archive/pdonnell-2020-08-29_06:13:10-multimds-wip-pdonnell-testing-20200829.021135-distro-basic-smithi/5387533 --owner scheduled_pdonnell@teuthology --block -- /tmp/teuthology-worker.7oulzk57.tmp
2020-08-29T11:01:22.316 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-08-29T11:01:22.378 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/pdonnell-2020-08-29_06:13:10-multimds-wip-pdonnell-testing-20200829.021135-distro-basic-smithi/5387533
  branch: wip-pdonnell-testing-20200829.021135
  description: multimds/basic/{0-supported-random-distro$/{rhel_8} begin clusters/9-mds
    conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable
    whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter
    tasks/cephfs_test_exports}
  email: pdonnell@redhat.com
  first_in_suite: false
  job_id: '5387533'
  kernel:
    kdb: true
    sha1: distro
  last_in_suite: false
  log-rotate:
    ceph-mds: 10G
    ceph-osd: 10G
  machine_type: smithi
  name: pdonnell-2020-08-29_06:13:10-multimds-wip-pdonnell-testing-20200829.021135-distro-basic-smithi
  nuke-on-error: true
  openstack:
  - volumes:
      count: 4
      size: 30
  - machine:
      disk: 200
  os_type: rhel
  os_version: '8.1'
  overrides:
    admin_socket:
      branch: wip-pdonnell-testing-20200829.021135
    ceph:
      conf:
        client:
          client mount timeout: 600
          debug client: 20
          debug ms: 1
          fuse default permissions: false
        mds:
          debug mds: 20
          debug ms: 1
          mds bal frag: true
          mds bal fragment size max: 10000
          mds bal merge size: 5
          mds bal split bits: 3
          mds bal split size: 100
          mds debug frag: true
          mds debug scatterstat: true
          mds op complaint time: 180
          mds verify scatter: true
          osd op complaint time: 180
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
          mon op complaint time: 120
        osd:
          bluestore block size: 96636764160
          bluestore compression mode: aggressive
          bluestore fsck on mount: true
          debug bluefs: 20
          debug bluestore: 20
          debug filestore: 20
          debug journal: 20
          debug ms: 20
          debug osd: 25
          debug rocksdb: 10
          mon osd backfillfull_ratio: 0.85
          mon osd full ratio: 0.9
          mon osd nearfull ratio: 0.8
          osd failsafe full ratio: 0.95
          osd objectstore: bluestore
          osd op complaint time: 180
      fs: xfs
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - overall HEALTH_
      - \(FS_DEGRADED\)
      - \(MDS_FAILED\)
      - \(MDS_DEGRADED\)
      - \(FS_WITH_FAILED_MDS\)
      - \(MDS_DAMAGE\)
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - \(FS_INLINE_DATA_DEPRECATED\)
      - overall HEALTH_
      - \(OSD_DOWN\)
      - \(OSD_
      - but it is still running
      - is not responding
      - Replacing daemon mds
      log-whitelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      max_mds: 9
      sha1: cbb55ee5e831579d31865bd96d4527993a598ddc
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon:
          osd default pool size: 2
    install:
      ceph:
        sha1: cbb55ee5e831579d31865bd96d4527993a598ddc
    selinux:
      whitelist:
      - scontext=system_u:system_r:logrotate_t:s0
    thrashosds:
      bdev_inject_crash: 2
      bdev_inject_crash_probability: 0.5
    workunit:
      branch: wip-pdonnell-testing-20200829.021135
      sha1: cbb55ee5e831579d31865bd96d4527993a598ddc
  owner: scheduled_pdonnell@teuthology
  priority: 97
  repo: git://git.ceph.com/ceph-ci.git
  roles:
  - - mon.a
    - mon.c
    - mgr.y
    - mds.a
    - mds.b
    - mds.c
    - mds.d
    - osd.0
    - osd.1
    - osd.2
    - osd.3
  - - mon.b
    - mgr.x
    - mds.e
    - mds.f
    - mds.g
    - mds.h
    - mds.i
    - osd.4
    - osd.5
    - osd.6
    - osd.7
  - - client.0
    - client.1
  sha1: cbb55ee5e831579d31865bd96d4527993a598ddc
  sleep_before_teardown: 0
  suite: multimds
  suite_branch: wip-pdonnell-testing-20200829.021135
  suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200829.021135/qa
  suite_relpath: qa
  suite_repo: git://git.ceph.com/ceph-ci.git
  suite_sha1: cbb55ee5e831579d31865bd96d4527993a598ddc
  tasks:
  - install:
      extra_packages:
        deb:
        - python3-cephfs
        - cephfs-shell
        rpm:
        - python3-cephfs
      extra_system_packages:
        deb:
        - bison
        - flex
        - libelf-dev
        - libssl-dev
        - network-manager
        - iproute2
        - util-linux
        - dump
        - indent
        - libaio-dev
        - libtool-bin
        - uuid-dev
        - xfslibs-dev
        rpm:
        - bison
        - flex
        - elfutils-libelf-devel
        - openssl-devel
        - NetworkManager
        - iproute
        - util-linux
        - libacl-devel
        - libaio-devel
        - libattr-devel
        - libtool
        - libuuid-devel
        - xfsdump
        - xfsprogs
        - xfsprogs-devel
        - libaio-devel
        - libtool
        - libuuid-devel
        - xfsprogs-devel
  - ceph: null
  - ceph-fuse: null
  - check-counter:
      counters:
        mds:
        - mds.exported
        - mds.imported
  - cephfs_test_runner:
      fail_on_skip: false
      modules:
      - tasks.cephfs.test_exports
  teuthology_branch: rpm-test
  timestamp: 2020-08-29_06:13:10
  tube: smithi
  user: pdonnell
  verbose: false
  worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.12120
2020-08-29T11:01:22.378 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200829.021135/qa; will attempt to use it
2020-08-29T11:01:22.379 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200829.021135/qa/tasks
2020-08-29T11:01:22.379 INFO:teuthology.run_tasks:Running task internal.check_packages...
2020-08-29T11:01:22.381 INFO:teuthology.task.internal:Checking packages...
2020-08-29T11:01:22.406 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'cbb55ee5e831579d31865bd96d4527993a598ddc'
2020-08-29T11:01:22.407 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2020-08-29T11:01:22.407 INFO:teuthology.packaging:ref: None
2020-08-29T11:01:22.408 INFO:teuthology.packaging:tag: None
2020-08-29T11:01:22.408 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20200829.021135
2020-08-29T11:01:22.408 INFO:teuthology.packaging:sha1: cbb55ee5e831579d31865bd96d4527993a598ddc
2020-08-29T11:01:22.409 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20200829.021135
2020-08-29T11:01:22.575 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-4877.gcbb55ee
2020-08-29T11:01:22.575 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2020-08-29T11:01:22.590 INFO:teuthology.task.internal:no buildpackages task found
2020-08-29T11:01:22.590 INFO:teuthology.run_tasks:Running task internal.lock_machines...
2020-08-29T11:01:22.616 INFO:teuthology.task.internal.lock_machines:Locking machines...
2020-08-29T11:01:22.617 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-08-29T11:01:22.673 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:01:32.699 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:01:42.716 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:01:52.736 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:02:02.754 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:02:12.771 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:02:22.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:02:32.812 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:02:42.839 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:02:52.860 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:03:02.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:03:12.903 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:03:22.921 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:03:32.943 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:03:42.963 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:03:52.983 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:04:02.999 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:04:13.020 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-08-29T11:04:23.041 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:04:33.060 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:04:43.077 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:04:53.097 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:05:03.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:05:13.134 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:05:23.150 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:05:33.168 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:05:43.188 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:05:53.208 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:06:03.228 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:06:13.248 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-08-29T11:06:23.268 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-08-29T11:06:33.287 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:06:43.303 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:06:53.324 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:07:03.342 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:07:13.362 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:07:23.377 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:07:33.395 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:07:43.413 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:07:53.435 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:08:03.454 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:08:13.475 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:08:23.495 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-08-29T11:08:33.518 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:08:43.539 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:08:53.557 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:09:03.577 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:09:13.596 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:09:23.613 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:09:33.632 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:09:43.651 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:09:53.669 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-08-29T11:10:03.695 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:10:13.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-08-29T11:10:23.734 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:10:33.768 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-08-29T11:10:43.785 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:10:53.802 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:11:03.820 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:11:13.842 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:11:23.860 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:11:33.882 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-08-29T11:11:43.899 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:11:53.919 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:12:03.941 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:12:13.963 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-08-29T11:12:23.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:12:34.011 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-08-29T11:12:44.027 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:12:54.042 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:13:04.062 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:13:14.082 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:13:24.103 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:13:34.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:13:44.138 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:13:54.157 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:14:04.178 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:14:14.196 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-08-29T11:14:24.216 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:14:34.235 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:14:44.251 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:14:54.267 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:15:04.284 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:15:14.305 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:15:24.336 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:15:34.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:15:44.381 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:15:54.408 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:16:04.431 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:16:14.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:16:24.468 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:16:34.494 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:16:44.517 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:16:54.536 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:17:04.559 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:17:14.579 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:17:24.599 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:17:34.617 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:17:44.653 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:17:54.673 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:18:04.701 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-08-29T11:18:14.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:18:24.738 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:18:34.758 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:18:44.776 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:18:54.794 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:19:04.811 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-08-29T11:19:14.830 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 3, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-08-29_06:13:10-multimds-wip-pdonnell-testing-20200829.021135-distro-basic-smithi/5387533'}
2020-08-29T11:19:14.855 DEBUG:teuthology.lock.ops:locked ubuntu@smithi196.front.sepia.ceph.com, ubuntu@smithi069.front.sepia.ceph.com, ubuntu@smithi110.front.sepia.ceph.com
2020-08-29T11:19:14.974 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fb8289439b0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fb8271b2f28>, signals=[15])
2020-08-29T11:19:15.249 INFO:teuthology.provision.fog.smithi196:Scheduling deploy of rhel 8.1
2020-08-29T11:19:15.253 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of rhel 8.1
2020-08-29T11:19:15.285 INFO:teuthology.provision.fog.smithi069:Scheduling deploy of rhel 8.1
2020-08-29T11:19:15.809 INFO:teuthology.orchestra.console:Power off smithi069
2020-08-29T11:19:15.811 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2020-08-29T11:19:15.831 INFO:teuthology.orchestra.console:Power off smithi110
2020-08-29T11:19:15.832 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2020-08-29T11:19:15.852 INFO:teuthology.orchestra.console:Power off smithi196
2020-08-29T11:19:15.853 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2020-08-29T11:19:15.865 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:15.877 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:15.889 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:16.001 INFO:teuthology.orchestra.console:Power off for smithi110 completed
2020-08-29T11:19:16.003 INFO:teuthology.orchestra.console:Power off for smithi069 completed
2020-08-29T11:19:16.012 INFO:teuthology.orchestra.console:Power off for smithi196 completed
2020-08-29T11:19:16.102 INFO:teuthology.orchestra.console:Power on smithi110
2020-08-29T11:19:16.103 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-08-29T11:19:16.116 INFO:teuthology.orchestra.console:Power on smithi069
2020-08-29T11:19:16.117 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-08-29T11:19:16.128 INFO:teuthology.orchestra.console:Power on smithi196
2020-08-29T11:19:16.129 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-08-29T11:19:16.142 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:16.152 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:16.162 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:20.176 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:20.188 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:20.201 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:20.403 INFO:teuthology.orchestra.console:Power on for smithi110 completed
2020-08-29T11:19:20.504 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish
2020-08-29T11:19:24.295 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:24.315 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-08-29T11:19:24.510 INFO:teuthology.orchestra.console:Power on for smithi069 completed
2020-08-29T11:19:24.529 INFO:teuthology.orchestra.console:Power on for smithi196 completed
2020-08-29T11:19:24.613 INFO:teuthology.provision.fog.smithi069:Waiting for deploy to finish
2020-08-29T11:19:24.630 INFO:teuthology.provision.fog.smithi196:Waiting for deploy to finish
2020-08-29T11:21:57.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:06.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:15.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:24.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:33.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:39.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:42.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:42.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:48.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:51.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:22:57.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:01.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:06.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:10.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:15.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:19.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:25.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:28.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:34.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:37.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:48.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:23:49.050 INFO:teuthology.orchestra.run:Running command with timeout 600
2020-08-29T11:23:49.053 INFO:teuthology.orchestra.run.smithi196:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2020-08-29T11:23:49.355 INFO:teuthology.provision.fog.smithi196:Node is ready
2020-08-29T11:23:49.355 INFO:teuthology.orchestra.run.smithi196:> hostname
2020-08-29T11:23:49.369 INFO:teuthology.orchestra.run.smithi196.stdout:smithi196.front.sepia.ceph.com
2020-08-29T11:23:49.370 INFO:teuthology.orchestra.run.smithi196:> grep smithi196.front.sepia.ceph.com /etc/hosts
2020-08-29T11:23:49.424 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.15.196 smithi196.front.sepia.ceph.com smithi196
2020-08-29T11:23:49.425 INFO:teuthology.orchestra.run.smithi196:> sudo hostname smithi196
2020-08-29T11:23:49.547 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hosts
2020-08-29T11:23:49.575 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/172.21.15.196/172.21.15.196/g' /etc/hosts
2020-08-29T11:23:49.643 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hostname
2020-08-29T11:23:49.709 INFO:teuthology.orchestra.run.smithi196:> sudo hostname smithi196
2020-08-29T11:23:49.775 INFO:teuthology.provision.fog.smithi196:Deploy complete!
2020-08-29T11:23:49.777 DEBUG:teuthology.parallel:result is None
2020-08-29T11:24:40.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:24:40.438 INFO:teuthology.orchestra.run:Running command with timeout 600
2020-08-29T11:24:40.439 INFO:teuthology.orchestra.run.smithi110:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2020-08-29T11:24:40.731 INFO:teuthology.provision.fog.smithi110:Node is ready
2020-08-29T11:24:40.732 INFO:teuthology.orchestra.run.smithi110:> hostname
2020-08-29T11:24:40.746 INFO:teuthology.orchestra.run.smithi110.stdout:smithi110.front.sepia.ceph.com
2020-08-29T11:24:40.747 INFO:teuthology.orchestra.run.smithi110:> grep smithi110.front.sepia.ceph.com /etc/hosts
2020-08-29T11:24:40.801 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.15.110 smithi110.front.sepia.ceph.com smithi110
2020-08-29T11:24:40.808 INFO:teuthology.orchestra.run.smithi110:> sudo hostname smithi110
2020-08-29T11:24:40.920 INFO:teuthology.orchestra.run.smithi110:> sudo sed -i -e 's/smithi110.front.sepia.ceph.com/smithi110/g' /etc/hosts
2020-08-29T11:24:40.949 INFO:teuthology.orchestra.run.smithi110:> sudo sed -i -e 's/172.21.15.110/172.21.15.110/g' /etc/hosts
2020-08-29T11:24:41.016 INFO:teuthology.orchestra.run.smithi110:> sudo sed -i -e 's/smithi110.front.sepia.ceph.com/smithi110/g' /etc/hostname
2020-08-29T11:24:41.083 INFO:teuthology.orchestra.run.smithi110:> sudo hostname smithi110
2020-08-29T11:24:41.149 INFO:teuthology.provision.fog.smithi110:Deploy complete!
2020-08-29T11:24:41.151 DEBUG:teuthology.parallel:result is None
2020-08-29T11:24:43.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:24:43.601 INFO:teuthology.orchestra.run:Running command with timeout 600
2020-08-29T11:24:43.601 INFO:teuthology.orchestra.run.smithi069:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2020-08-29T11:24:43.973 INFO:teuthology.provision.fog.smithi069:Node is ready
2020-08-29T11:24:43.974 INFO:teuthology.orchestra.run.smithi069:> hostname
2020-08-29T11:24:43.988 INFO:teuthology.orchestra.run.smithi069.stdout:smithi069.front.sepia.ceph.com
2020-08-29T11:24:43.989 INFO:teuthology.orchestra.run.smithi069:> grep smithi069.front.sepia.ceph.com /etc/hosts
2020-08-29T11:24:44.043 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69 smithi069.front.sepia.ceph.com smithi069
2020-08-29T11:24:44.045 INFO:teuthology.orchestra.run.smithi069:> sudo hostname smithi069
2020-08-29T11:24:44.157 INFO:teuthology.orchestra.run.smithi069:> sudo sed -i -e 's/smithi069.front.sepia.ceph.com/smithi069/g' /etc/hosts
2020-08-29T11:24:44.185 INFO:teuthology.orchestra.run.smithi069:> sudo sed -i -e 's/172.21.15.69/172.21.15.69/g' /etc/hosts
2020-08-29T11:24:44.259 INFO:teuthology.orchestra.run.smithi069:> sudo sed -i -e 's/smithi069.front.sepia.ceph.com/smithi069/g' /etc/hostname
2020-08-29T11:24:44.319 INFO:teuthology.orchestra.run.smithi069:> sudo hostname smithi069
2020-08-29T11:24:44.385 INFO:teuthology.provision.fog.smithi069:Deploy complete!
2020-08-29T11:24:44.386 DEBUG:teuthology.parallel:result is None
2020-08-29T11:24:44.589 INFO:teuthology.lock.ops:Checking smithi196.front.sepia.ceph.com
2020-08-29T11:24:44.590 INFO:teuthology.lock.ops:New key found. Updating...
2020-08-29T11:24:44.608 INFO:teuthology.lock.ops:Checking smithi069.front.sepia.ceph.com
2020-08-29T11:24:44.608 INFO:teuthology.lock.ops:New key found. Updating...
2020-08-29T11:24:44.621 INFO:teuthology.lock.ops:Checking smithi110.front.sepia.ceph.com
2020-08-29T11:24:44.621 INFO:teuthology.lock.ops:New key found. Updating...
2020-08-29T11:24:44.637 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: set os type and version on server
2020-08-29T11:24:44.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-08-29T11:24:44.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:24:44.715 INFO:teuthology.orchestra.run.smithi196:> true
2020-08-29T11:24:44.880 INFO:teuthology.orchestra.run.smithi196:> uname -m
2020-08-29T11:24:44.934 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64
2020-08-29T11:24:44.935 INFO:teuthology.orchestra.run.smithi196:> cat /etc/os-release
2020-08-29T11:24:44.988 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Red Hat Enterprise Linux"
2020-08-29T11:24:44.988 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="8.1 (Ootpa)"
2020-08-29T11:24:44.989 INFO:teuthology.orchestra.run.smithi196.stdout:ID="rhel"
2020-08-29T11:24:44.989 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="fedora"
2020-08-29T11:24:44.989 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="8.1"
2020-08-29T11:24:44.989 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el8"
2020-08-29T11:24:44.989 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)"
2020-08-29T11:24:44.990 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31"
2020-08-29T11:24:44.990 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA"
2020-08-29T11:24:44.990 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.redhat.com/"
2020-08-29T11:24:44.990 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2020-08-29T11:24:44.990 INFO:teuthology.orchestra.run.smithi196.stdout:
2020-08-29T11:24:44.991 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2020-08-29T11:24:44.991 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.1
2020-08-29T11:24:44.991 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2020-08-29T11:24:44.991 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.1"
2020-08-29T11:24:44.993 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server
2020-08-29T11:24:45.007 INFO:teuthology.lock.ops:Updating [smithi069.front.sepia.ceph.com]: set os type and version on server
2020-08-29T11:24:45.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-08-29T11:24:45.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:24:45.075 INFO:teuthology.orchestra.run.smithi069:> true
2020-08-29T11:24:45.232 INFO:teuthology.orchestra.run.smithi069:> uname -m
2020-08-29T11:24:45.286 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64
2020-08-29T11:24:45.286 INFO:teuthology.orchestra.run.smithi069:> cat /etc/os-release
2020-08-29T11:24:45.340 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="Red Hat Enterprise Linux"
2020-08-29T11:24:45.340 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="8.1 (Ootpa)"
2020-08-29T11:24:45.340 INFO:teuthology.orchestra.run.smithi069.stdout:ID="rhel"
2020-08-29T11:24:45.341 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="fedora"
2020-08-29T11:24:45.341 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="8.1"
2020-08-29T11:24:45.341 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el8"
2020-08-29T11:24:45.341 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)"
2020-08-29T11:24:45.341 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31"
2020-08-29T11:24:45.342 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA"
2020-08-29T11:24:45.342 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://www.redhat.com/"
2020-08-29T11:24:45.342 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2020-08-29T11:24:45.342 INFO:teuthology.orchestra.run.smithi069.stdout:
2020-08-29T11:24:45.342 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2020-08-29T11:24:45.342 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.1
2020-08-29T11:24:45.343 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2020-08-29T11:24:45.343 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.1"
2020-08-29T11:24:45.344 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server
2020-08-29T11:24:45.358 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: set os type and version on server
2020-08-29T11:24:45.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-08-29T11:24:45.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-08-29T11:24:45.426 INFO:teuthology.orchestra.run.smithi110:> true
2020-08-29T11:24:45.593 INFO:teuthology.orchestra.run.smithi110:> uname -m
2020-08-29T11:24:45.647 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64
2020-08-29T11:24:45.647 INFO:teuthology.orchestra.run.smithi110:> cat /etc/os-release
2020-08-29T11:24:45.701 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Red Hat Enterprise Linux"
2020-08-29T11:24:45.702 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="8.1 (Ootpa)"
2020-08-29T11:24:45.702 INFO:teuthology.orchestra.run.smithi110.stdout:ID="rhel"
2020-08-29T11:24:45.702 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="fedora"
2020-08-29T11:24:45.702 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="8.1"
2020-08-29T11:24:45.703 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el8"
2020-08-29T11:24:45.703 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.1 (Ootpa)"
2020-08-29T11:24:45.703 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31"
2020-08-29T11:24:45.703 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.1:GA"
2020-08-29T11:24:45.703 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.redhat.com/"
2020-08-29T11:24:45.704 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2020-08-29T11:24:45.704 INFO:teuthology.orchestra.run.smithi110.stdout:
2020-08-29T11:24:45.704 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2020-08-29T11:24:45.704 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.1
2020-08-29T11:24:45.704 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2020-08-29T11:24:45.705 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.1"
2020-08-29T11:24:45.705 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2020-08-29T11:24:45.728 INFO:teuthology.task.internal.lock_machines:3 smithi machines locked this try, 3/3 locked so far
2020-08-29T11:24:45.780 INFO:teuthology.task.internal.lock_machines:Locked targets:
  smithi069.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnFGagsrwGznHXn5hnrkVLOmD3U3W0FKhGQddd9c58Ctj8fFXmh7INID6keSrKAvBAnP/PSUJdinn0/zeZ3BTH+WIpWajjQ8IPCQPgTe2h+C1lmJpQpRWatqw1pcJDYW4xBiMSRmagHbi40znqXjHP9sHhtfWRFaL7lRpZYSwLnzDR8pzJoKVYxj14OlTZ+25pIdv3HxLBAVNlyPUxlsnJO2sZYB5NsA/1RsXaSyWh7sIaXPikHiXBCTl6YbkXzSvSAXEna6QRfo3uzP5bUwNr8JwimDaVk4eHDVm0mjc/ZZQNYOM7Pm1RI9nbt82pPUcFruyC7lfICmzHJlqO7P4jahl3DNEzpki9UnBv6i6wT2aMhkQGv6gGay6tBu6jcN4jCB0wyuPfPWUHzFSeDFYulBzv3VJ/k3WXtBvr0BKUIOva3LUdbZuTi220rK/SqbJznSyGrBDijcv8vSPOXWppsadE7CSgNwo57t6yiYiuU3ixXDVTBbAKyjvxBSvdqN0=
  smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqDPCuzENMnjoaNmyjeNEp1ABQsp7WxbirZhtWu41jb3XyOIKYNBidm50xHKKOfEprFPu69rT9lp/6SupKjk5WDq7Ew2a5U6B1+fmsjLt75VpLKAu/ee2cnbnOFUkQ5DbKk+paY72mvIn+nRO6pwp2B33lwg1mSBmx0HQSOOpP3UZzz1jivWhzZCVtZlqs/qC1Vpgjp2fyJ03yjncX2upuEcVZUyzPZdxacv0ef1UChEs3pZrsrpZIlZSeo94aC3YilZwQt9snQIMlm7evujz0YFVpR2MqnysNLaHYa0WUL2plO/rTKzkySQI8IQ6l1BA5mMo0Oz6Klb5JGsMjZrsmmNUxXRze8QiSu0o+KsuIfGpgpkFfidH6fVT2bn2apQ+am8QjVktlduEMiaRhczhRnvV8A4gFIvP6/wRogwXq/PIyeMl0TOHfKPBceODkHonykLN/yAoD6Fz+6yAxLyVXUTvo9tWMSdhKHjb42OxJOJEsprj4HpxlczNa72QIDus=
  smithi196.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4qtPRMu6FGxSgDGG0FQe4YB+lJ/K/5l0Wi4uBtNG6jlf8jXOmxZIVIK/ZvrQceMFu0M532wS7WC+VQeepU5o0V/2ajbJ2r8Ro/FRMN2ZBgTdgYKMFofnhU+x56gphh3Ubt9YSpV3vU2hiObLK2/5KPYVKP7IObdS7wDSeYDSkezq4ThTXe25V84tPNF9OkwcIQBCpW0tdF+x5dy2nKUvSqF+TEWFoNp6ICm9Fnwwv4dD2iZ29QvyfqJyEGRetfWN7/HdGsTBiwX+jhWcESpXvvdbEXtTbD760IE6lb2QeLP4FFzxVQ4HLofiY0E1EV0xerZKwMZpqHS0Oa8y2e0IFMuisrNmlTknhDiRon88i3Q0YMLGolqAjLOzRLUR7KfK5twr08vfUsYsLriVgMIpRd4TewgqCOXNcuSY3ZUBfmDcsAD3hObDNEFsJGYCB/GtG/rQp82yWxcIJHxqmz/R73GwC3Lveyx879g8vfdNVDYGKMRdlaC8s0oOYpE5nhh8=
2020-08-29T11:24:45.781 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-08-29T11:24:45.848 INFO:teuthology.run_tasks:Running task internal.save_config...
2020-08-29T11:24:45.857 INFO:teuthology.task.internal:Saving configuration
2020-08-29T11:24:45.879 INFO:teuthology.run_tasks:Running task internal.check_lock...
2020-08-29T11:24:46.017 INFO:teuthology.task.internal.check_lock:Checking locks...
2020-08-29T11:24:46.035 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-08-29 11:19:14.836533', 'description': '/home/teuthworker/archive/pdonnell-2020-08-29_06:13:10-multimds-wip-pdonnell-testing-20200829.021135-distro-basic-smithi/5387533', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKv3UHI1+pGk65uAmJGWyj/BtfY76MEfV0bTQhBX/xZ2ItsUdkANeOrx7Raq0WO8AyallhqlL/A9kUJtHAYvB9k=', 'os_version': '8.1', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi196.front.sepia.ceph.com'}
2020-08-29T11:24:46.053 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-08-29 11:19:14.836553', 'description': '/home/teuthworker/archive/pdonnell-2020-08-29_06:13:10-multimds-wip-pdonnell-testing-20200829.021135-distro-basic-smithi/5387533', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOaAvDnrfxoBWC9LT41XpnwsP3Kc28xd57KRtnJErRAOSh71QGC8hAAjIW1ic5dJYrCjCrAefIf5y9lX5VTH9AU=', 'os_version': '8.1', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi069.front.sepia.ceph.com'}
2020-08-29T11:24:46.068 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-08-29 11:19:14.836569', 'description': '/home/teuthworker/archive/pdonnell-2020-08-29_06:13:10-multimds-wip-pdonnell-testing-20200829.021135-distro-basic-smithi/5387533', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBJnAaFHVKJ0NDxT+XRDbZWG9pD9txX2GYiLM3Vc6l77zwllrcR4K0FKGsg+dO5Xn9LzHkLK2wCzBksgVKpSppM=', 'os_version': '8.1', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi110.front.sepia.ceph.com'}
2020-08-29T11:24:46.069 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2020-08-29T11:24:46.078 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'mds.d', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
2020-08-29T11:24:46.079 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'mds.e', 'mds.f', 'mds.g', 'mds.h', 'mds.i', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2020-08-29T11:24:46.079 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['client.0', 'client.1']
2020-08-29T11:24:46.079 INFO:teuthology.run_tasks:Running task console_log...
2020-08-29T11:24:46.175 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/teuthology/run_tasks.py", line 93, in run_tasks
    manager.__enter__()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/teuthology/task/__init__.py", line 123, in __enter__
    self.begin()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/teuthology/task/console_log.py", line 67, in begin
    self.start_logging()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/teuthology/task/console_log.py", line 75, in start_logging
    proc = remote.console.spawn_sol_log(log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/teuthology/orchestra/console.py", line 308, in spawn_sol_log
    proc = start()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/teuthology/orchestra/console.py", line 305, in start
    env=os.environ,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/virtualenv/lib/python3.6/site-packages/psutil/__init__.py", line 1430, in __init__
    self.__subproc = subprocess.Popen(*args, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/virtualenv/bin/python': '/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/virtualenv/bin/python'