2020-11-27T14:33:08.793 INFO:root:teuthology version: 1.0.0-e08e33e
2020-11-27T14:33:08.797 DEBUG:teuthology.run:Teuthology command: teuthology --block --name kchai-2020-11-27_14:17:32-rados-wip-kefu-testing-2020-11-27-2029-distro-basic-smithi --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2020-11-27_14:17:32-rados-wip-kefu-testing-2020-11-27-2029-distro-basic-smithi/5660221 --lock --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read} --verbose -- /tmp/teuthology-worker.1dp4dtr8.tmp
2020-11-27T14:33:08.850 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-11-27T14:33:08.923 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/kchai-2020-11-27_14:17:32-rados-wip-kefu-testing-2020-11-27-2029-distro-basic-smithi/5660221
  branch: wip-kefu-testing-2020-11-27-2029
  description: rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast
    mon_election/classic msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{default}
    supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read}
  email: tchaikov@gmail.com
  first_in_suite: false
  job_id: '5660221'
  kernel:
    kdb: true
    sha1: distro
  last_in_suite: false
  machine_type: smithi
  name: kchai-2020-11-27_14:17:32-rados-wip-kefu-testing-2020-11-27-2029-distro-basic-smithi
  nuke-on-error: true
  openstack:
  - volumes:
      count: 4
      size: 10
  - volumes:
      count: 4
      size: 30
  os_type: rhel
  os_version: '8.3'
  overrides:
    admin_socket:
      branch: wip-kefu-testing-2020-11-27-2029
    ceph:
      conf:
        global:
          mon client directed command retry: 5
          mon election default strategy: 1
          ms inject socket failures: 5000
          osd pool default ec fast read: true
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
          mon scrub interval: 300
        osd:
          debug filestore: 20
          debug journal: 20
          debug ms: 20
          debug osd: 25
          osd debug reject backfill probability: 0.3
          osd debug verify cached snaps: true
          osd debug verify missing on start: true
          osd heartbeat use min delay socket: true
          osd max backfills: 2
          osd max markdown count: 1000
          osd objectstore: filestore
          osd op queue: debug_random
          osd op queue cut off: debug_random
          osd scrub max interval: 120
          osd scrub min interval: 60
          osd shutdown pgref assert: true
          osd sloppy crc: true
      create_rbd_pool: false
      fs: xfs
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - \(OSD_SLOW_PING_TIME
      - but it is still running
      - objects unfound and apparently lost
      - overall HEALTH_
      - \(OSDMAP_FLAGS\)
      - \(OSD_
      - \(PG_
      - \(POOL_
      - \(CACHE_POOL_
      - \(SMALLER_PGP_NUM\)
      - \(OBJECT_
      - \(SLOW_OPS\)
      - \(REQUEST_SLOW\)
      - \(TOO_FEW_PGS\)
      - slow request
      log-whitelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      pre-mgr-commands:
      - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force
      sha1: ca3942f8ee575a70bca909217fa98ae434d67790
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon:
          osd default pool size: 2
        osd:
          osd objectstore: filestore
          osd sloppy crc: true
      filestore: true
      fs: xfs
    install:
      ceph:
        sha1: ca3942f8ee575a70bca909217fa98ae434d67790
    selinux:
      whitelist:
      - scontext=system_u:system_r:logrotate_t:s0
    workunit:
      branch: wip-kefu-testing-2020-11-27-2029
      sha1: ca3942f8ee575a70bca909217fa98ae434d67790
  owner: scheduled_kchai@teuthology
  priority: 70
  repo: git://git.ceph.com/ceph-ci.git
  roles:
  - - mon.a
    - mon.c
    - mgr.y
    - osd.0
    - osd.1
    - osd.2
    - osd.3
    - client.0
    - node-exporter.a
  - - mon.b
    - mgr.x
    - osd.4
    - osd.5
    - osd.6
    - osd.7
    - client.1
    - prometheus.a
    - node-exporter.b
  sha1: ca3942f8ee575a70bca909217fa98ae434d67790
  sleep_before_teardown: 0
  suite: rados
  suite_branch: wip-kefu-testing-2020-11-27-2029
  suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-27-2029/qa
  suite_relpath: qa
  suite_repo: git://git.ceph.com/ceph-ci.git
  suite_sha1: ca3942f8ee575a70bca909217fa98ae434d67790
  tasks:
  - install: null
  - ceph: null
  - thrashosds:
      chance_test_min_size: 3
      timeout: 1200
  - rados:
      clients:
      - client.0
      ec_pool: true
      fast_read: true
      max_in_flight: 64
      max_seconds: 600
      objects: 1024
      op_weights:
        append: 100
        copy_from: 50
        delete: 50
        read: 100
        rmattr: 25
        rollback: 50
        setattr: 25
        snap_create: 50
        snap_remove: 50
        write: 0
      ops: 400000
      size: 16384
  teuthology_branch: wip-task-archive-gzip
  timestamp: 2020-11-27_14:17:32
  tube: smithi
  user: kchai
  verbose: false
  worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.25221
2020-11-27T14:33:08.924 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-27-2029/qa; will attempt to use it
2020-11-27T14:33:08.925 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2020-11-27-2029/qa/tasks
2020-11-27T14:33:08.926 INFO:teuthology.run_tasks:Running task internal.check_packages...
2020-11-27T14:33:08.929 INFO:teuthology.task.internal:Checking packages...
2020-11-27T14:33:08.956 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'ca3942f8ee575a70bca909217fa98ae434d67790'
2020-11-27T14:33:08.957 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2020-11-27T14:33:08.957 INFO:teuthology.packaging:ref: None
2020-11-27T14:33:08.958 INFO:teuthology.packaging:tag: None
2020-11-27T14:33:08.959 INFO:teuthology.packaging:branch: wip-kefu-testing-2020-11-27-2029
2020-11-27T14:33:08.959 INFO:teuthology.packaging:sha1: ca3942f8ee575a70bca909217fa98ae434d67790
2020-11-27T14:33:08.960 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2020-11-27-2029
2020-11-27T14:33:09.134 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7610.gca3942f8
2020-11-27T14:33:09.134 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2020-11-27T14:33:09.140 INFO:teuthology.task.internal:no buildpackages task found
2020-11-27T14:33:09.140 INFO:teuthology.run_tasks:Running task internal.lock_machines...
2020-11-27T14:33:09.148 INFO:teuthology.task.internal.lock_machines:Locking machines...
2020-11-27T14:33:09.148 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-11-27T14:33:09.213 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:33:19.231 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)...
2020-11-27T14:33:29.246 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:33:39.262 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:33:49.279 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:33:59.294 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:34:09.311 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:34:19.323 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:34:29.333 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:34:39.343 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:34:49.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:34:59.373 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:35:09.389 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)...
2020-11-27T14:35:19.403 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)...
2020-11-27T14:35:29.424 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)...
2020-11-27T14:35:39.443 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2020-11-27_14:17:32-rados-wip-kefu-testing-2020-11-27-2029-distro-basic-smithi/5660221'}
2020-11-27T14:35:39.462 ERROR:teuthology.lock.ops:Insufficient nodes available to lock 2 smithi nodes.
2020-11-27T14:35:39.462 ERROR:teuthology.lock.ops:{"message": "only 0 nodes available"}
2020-11-27T14:35:39.462 INFO:teuthology.task.internal.lock_machines:0 smithi machines locked this try, 0/2 locked so far
2020-11-27T14:35:39.463 INFO:teuthology.task.internal.lock_machines:0 machines locked (0 new); need 2 more
2020-11-27T14:35:39.463 WARNING:teuthology.task.internal.lock_machines:Could not lock enough machines, waiting...
2020-11-27T14:35:49.480 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:35:59.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:36:09.515 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:36:19.533 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:36:29.551 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:36:39.566 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:36:49.582 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)...
2020-11-27T14:36:59.598 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:37:09.608 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:37:19.642 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:37:29.658 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:37:39.669 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:37:49.687 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)...
2020-11-27T14:37:59.702 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_kchai@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/kchai-2020-11-27_14:17:32-rados-wip-kefu-testing-2020-11-27-2029-distro-basic-smithi/5660221'}
2020-11-27T14:37:59.723 DEBUG:teuthology.lock.ops:locked ubuntu@smithi027.front.sepia.ceph.com, ubuntu@smithi196.front.sepia.ceph.com
2020-11-27T14:37:59.778 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f7176e14e48>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f71757dd8c8>, signals=[15])
2020-11-27T14:37:59.993 INFO:teuthology.provision.fog.smithi027:Scheduling deploy of rhel 8.3
2020-11-27T14:38:00.000 INFO:teuthology.provision.fog.smithi196:Scheduling deploy of rhel 8.3
2020-11-27T14:38:00.364 INFO:teuthology.orchestra.console:Power off smithi196
2020-11-27T14:38:00.365 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2020-11-27T14:38:00.379 INFO:teuthology.orchestra.console:Power off smithi027
2020-11-27T14:38:00.381 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2020-11-27T14:38:00.392 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-27T14:38:00.403 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-27T14:38:00.517 INFO:teuthology.orchestra.console:Power off for smithi196 completed
2020-11-27T14:38:00.526 INFO:teuthology.orchestra.console:Power off for smithi027 completed
2020-11-27T14:38:00.619 INFO:teuthology.orchestra.console:Power on smithi196
2020-11-27T14:38:00.619 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-11-27T14:38:00.629 INFO:teuthology.orchestra.console:Power on smithi027
2020-11-27T14:38:00.630 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-11-27T14:38:00.645 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-27T14:38:00.655 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-27T14:38:04.666 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-27T14:38:04.677 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-27T14:38:08.778 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-27T14:38:08.788 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-27T14:38:08.999 INFO:teuthology.orchestra.console:Power on for smithi027 completed
2020-11-27T14:38:09.023 INFO:teuthology.orchestra.console:Power on for smithi196 completed
2020-11-27T14:38:09.100 INFO:teuthology.provision.fog.smithi027:Waiting for deploy to finish
2020-11-27T14:38:09.125 INFO:teuthology.provision.fog.smithi196:Waiting for deploy to finish
2020-11-27T14:40:25.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:40:34.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:40:40.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:40:43.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:40:49.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:40:52.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:40:58.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:41:01.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:41:07.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:41:10.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:41:16.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:41:19.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:41:25.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:41:28.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:41:34.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:41:37.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:42:40.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:42:41.015 INFO:teuthology.orchestra.run:Running command with timeout 600
2020-11-27T14:42:41.017 INFO:teuthology.orchestra.run.smithi196:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2020-11-27T14:42:41.359 INFO:teuthology.provision.fog.smithi196:Node is ready
2020-11-27T14:42:41.359 INFO:teuthology.orchestra.run.smithi196:> hostname
2020-11-27T14:42:41.373 INFO:teuthology.orchestra.run.smithi196.stdout:smithi196.front.sepia.ceph.com
2020-11-27T14:42:41.373 INFO:teuthology.orchestra.run.smithi196:> grep smithi196.front.sepia.ceph.com /etc/hosts
2020-11-27T14:42:41.427 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.15.196 smithi196.front.sepia.ceph.com smithi196
2020-11-27T14:42:41.428 INFO:teuthology.orchestra.run.smithi196:> sudo hostname smithi196
2020-11-27T14:42:41.535 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hosts
2020-11-27T14:42:41.559 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/172.21.15.196/172.21.15.196/g' /etc/hosts
2020-11-27T14:42:41.622 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hostname
2020-11-27T14:42:41.687 INFO:teuthology.orchestra.run.smithi196:> sudo hostname smithi196
2020-11-27T14:42:41.750 INFO:teuthology.provision.fog.smithi196:Deploy complete!
2020-11-27T14:42:41.751 DEBUG:teuthology.parallel:result is None
2020-11-27T14:42:43.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:42:43.945 INFO:teuthology.orchestra.run:Running command with timeout 600
2020-11-27T14:42:43.945 INFO:teuthology.orchestra.run.smithi027:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2020-11-27T14:42:44.240 INFO:teuthology.provision.fog.smithi027:Node is ready
2020-11-27T14:42:44.241 INFO:teuthology.orchestra.run.smithi027:> hostname
2020-11-27T14:42:44.256 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027.front.sepia.ceph.com
2020-11-27T14:42:44.257 INFO:teuthology.orchestra.run.smithi027:> grep smithi027.front.sepia.ceph.com /etc/hosts
2020-11-27T14:42:44.311 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27 smithi027.front.sepia.ceph.com smithi027
2020-11-27T14:42:44.311 INFO:teuthology.orchestra.run.smithi027:> sudo hostname smithi027
2020-11-27T14:42:44.425 INFO:teuthology.orchestra.run.smithi027:> sudo sed -i -e 's/smithi027.front.sepia.ceph.com/smithi027/g' /etc/hosts
2020-11-27T14:42:44.451 INFO:teuthology.orchestra.run.smithi027:> sudo sed -i -e 's/172.21.15.27/172.21.15.27/g' /etc/hosts
2020-11-27T14:42:44.516 INFO:teuthology.orchestra.run.smithi027:> sudo sed -i -e 's/smithi027.front.sepia.ceph.com/smithi027/g' /etc/hostname
2020-11-27T14:42:44.581 INFO:teuthology.orchestra.run.smithi027:> sudo hostname smithi027
2020-11-27T14:42:44.645 INFO:teuthology.provision.fog.smithi027:Deploy complete!
2020-11-27T14:42:44.646 DEBUG:teuthology.parallel:result is None
2020-11-27T14:42:44.778 INFO:teuthology.lock.ops:Checking smithi027.front.sepia.ceph.com
2020-11-27T14:42:44.779 INFO:teuthology.lock.ops:New key found. Updating...
2020-11-27T14:42:44.796 INFO:teuthology.lock.ops:Checking smithi196.front.sepia.ceph.com
2020-11-27T14:42:44.796 INFO:teuthology.lock.ops:New key found. Updating...
2020-11-27T14:42:44.812 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: set os type and version on server
2020-11-27T14:42:44.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-11-27T14:42:44.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:42:44.882 INFO:teuthology.orchestra.run.smithi027:> true
2020-11-27T14:42:45.048 INFO:teuthology.orchestra.run.smithi027:> uname -m
2020-11-27T14:42:45.103 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64
2020-11-27T14:42:45.104 INFO:teuthology.orchestra.run.smithi027:> cat /etc/os-release
2020-11-27T14:42:45.157 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="Red Hat Enterprise Linux"
2020-11-27T14:42:45.158 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8.3 (Ootpa)"
2020-11-27T14:42:45.158 INFO:teuthology.orchestra.run.smithi027.stdout:ID="rhel"
2020-11-27T14:42:45.158 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="fedora"
2020-11-27T14:42:45.158 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8.3"
2020-11-27T14:42:45.159 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8"
2020-11-27T14:42:45.159 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2020-11-27T14:42:45.159 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31"
2020-11-27T14:42:45.160 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2020-11-27T14:42:45.160 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://www.redhat.com/"
2020-11-27T14:42:45.160 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2020-11-27T14:42:45.160 INFO:teuthology.orchestra.run.smithi027.stdout:
2020-11-27T14:42:45.160 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2020-11-27T14:42:45.161 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2020-11-27T14:42:45.161 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2020-11-27T14:42:45.161 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2020-11-27T14:42:45.162 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2020-11-27T14:42:45.179 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: set os type and version on server
2020-11-27T14:42:45.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-11-27T14:42:45.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-27T14:42:45.246 INFO:teuthology.orchestra.run.smithi196:> true
2020-11-27T14:42:45.413 INFO:teuthology.orchestra.run.smithi196:> uname -m
2020-11-27T14:42:45.467 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64
2020-11-27T14:42:45.468 INFO:teuthology.orchestra.run.smithi196:> cat /etc/os-release
2020-11-27T14:42:45.521 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Red Hat Enterprise Linux"
2020-11-27T14:42:45.521 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="8.3 (Ootpa)"
2020-11-27T14:42:45.522 INFO:teuthology.orchestra.run.smithi196.stdout:ID="rhel"
2020-11-27T14:42:45.522 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="fedora"
2020-11-27T14:42:45.522 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="8.3"
2020-11-27T14:42:45.522 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el8"
2020-11-27T14:42:45.523 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2020-11-27T14:42:45.523 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31"
2020-11-27T14:42:45.523 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2020-11-27T14:42:45.523 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.redhat.com/"
2020-11-27T14:42:45.524 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2020-11-27T14:42:45.524 INFO:teuthology.orchestra.run.smithi196.stdout:
2020-11-27T14:42:45.524 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2020-11-27T14:42:45.524 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2020-11-27T14:42:45.525 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2020-11-27T14:42:45.525 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2020-11-27T14:42:45.525 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server
2020-11-27T14:42:45.542 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far
2020-11-27T14:42:45.579 INFO:teuthology.task.internal.lock_machines:Locked targets:
  smithi027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC45Ef5TdUaJECSyEx2h0zveb9uyQD2OouC3t+sXveFDG4aE2R506aYbKIDJ9xslb6LELGLUt0nGrF+dGbwHShhni1IxUokwXA3mPZUc7hhfKND3GazY5wIws6Zj+3vgVE769goOAbbwHAABDfxoJwgkn7FgLNsZKdabAOPuiMF6vE+qHp0SfAwD/W/MPdJfZ1MiAmRKeGUMiRT+QwECRLh3WiLBNqApFwu31sptPnwF7BhCVQQp3Zj0XaQGcZDEt7dZ5WSnwkQhpqEvvl/Qe7FObFB5O2gEMETTmGOLCaxxE3MCJndqpFMSdgApHhyptQixEPTf01AN21D1w7RIjtHjD/qr4PjnNNR27cdG/G8CRQAqotfVQexPzIYXMXFP9u0juYdodU5JutPLQTDqiHQX9MBRcmqfntb1k4eCZ1V7GISq5v2GG8C47EUMic6q1X+8d/lZmPvuwErtXAs2MDTMQlJFHJsDGM1SROkCLZ1uQpsyG3FFAoF4mbM49ZOO10=
  smithi196.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC56vqN6e04VtYuSl+l9PFoyeIDpt++7hGll5cktKDvckwdEGRpEbVhwbgB974jgdRQrzSqpmlOoW6t8H7hAw66KQGp1xyEh9QDTOQdSvFExIiqHqwRRW9ZFOXW6dfXGLaMmmaxY9ulbHlzK8llIjdgaYtkAEhXZh7Qo8Zhd4x9dNPOixYsUkxpXBV8Fm1BZhI0xHPQLlFmwyrH1Nnmj94gG03DwHbWLktRaQ0HHA7B3IGh8Inh9ewBfwa0QHzyS8Z98e0Ma0KwYcbbOo+tC47gd5S3v55JgBpgdmW+z+zTlSUNUFSNDnIhu5j4YhBHxo0wEX+jLKGbFmGmSIwBwGrBPv7ViLBqr3t2XGVwht6nuXq+d8FVhkuqBZmhZrN+L/SfjihbkypqBQnbaAUOMiBbuDQX6YSuDJl0oNp0RZZzfIOBYZF2Rgp/mnCn9xpDKDCghJLrBlG5mPdwQ7Y0HkyU4RfggPfYhXinTLOYDygql5rOLoJAyprdL9hV2Ixnbnc=
2020-11-27T14:42:45.581 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-11-27T14:42:45.636 INFO:teuthology.run_tasks:Running task internal.save_config...
2020-11-27T14:42:45.646 INFO:teuthology.task.internal:Saving configuration
2020-11-27T14:42:45.663 INFO:teuthology.run_tasks:Running task internal.check_lock...
2020-11-27T14:42:45.673 INFO:teuthology.task.internal.check_lock:Checking locks...
2020-11-27T14:42:45.692 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-27 14:37:59.711101', 'description': '/home/teuthworker/archive/kchai-2020-11-27_14:17:32-rados-wip-kefu-testing-2020-11-27-2029-distro-basic-smithi/5660221', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCAjvaioXbNbz+JZViVBOBVQIL3kjQIGIGPzTHIaWHhcD42LwyzCEZL9N8otenhhBegoFAkkWKyWtd5LzGJRisY=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi027.front.sepia.ceph.com'}
2020-11-27T14:42:45.708 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-27 14:37:59.711122', 'description': '/home/teuthworker/archive/kchai-2020-11-27_14:17:32-rados-wip-kefu-testing-2020-11-27-2029-distro-basic-smithi/5660221', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKsQsVcknyQO9xYs32pLC6oCXMaHkEpbqOnWcJoXCTC2IaQcKpij4g0jkE2EmhjSWOn401STQrItNtdpu4xux0E=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi196.front.sepia.ceph.com'}
2020-11-27T14:42:45.709 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2020-11-27T14:42:45.720 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2020-11-27T14:42:45.720 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2020-11-27T14:42:45.720 INFO:teuthology.run_tasks:Running task console_log...
2020-11-27T14:42:45.782 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-task-archive-gzip/teuthology/run_tasks.py", line 93, in run_tasks
    manager.__enter__()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-task-archive-gzip/teuthology/task/__init__.py", line 123, in __enter__
    self.begin()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-task-archive-gzip/teuthology/task/console_log.py", line 67, in begin
    self.start_logging()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-task-archive-gzip/teuthology/task/console_log.py", line 75, in start_logging
    proc = remote.console.spawn_sol_log(log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-task-archive-gzip/teuthology/orchestra/console.py", line 308, in spawn_sol_log
    proc = start()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-task-archive-gzip/teuthology/orchestra/console.py", line 305, in start
    env=os.environ,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-task-archive-gzip/virtualenv/lib/python3.6/site-packages/psutil/__init__.py", line 1430, in __init__
    self.__subproc = subprocess.Popen(*args, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/teuthworker/src/git.ceph.com_git_teuthology_wip-task-archive-gzip/virtualenv/bin/python': '/home/teuthworker/src/git.ceph.com_git_teuthology_wip-task-archive-gzip/virtualenv/bin/python'