2020-11-19T21:42:42.602 INFO:root:teuthology version: 1.0.0-357fd4a
2020-11-19T21:42:42.604 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2020-11-19_17:21:50-rados-wip-yuri4-testing-2020-11-19-0746-nautilus-distro-basic-smithi --verbose --archive /home/teuthworker/archive/yuriw-2020-11-19_17:21:50-rados-wip-yuri4-testing-2020-11-19-0746-nautilus-distro-basic-smithi/5637915 --owner scheduled_yuriw@teuthology --lock --block --description rados/multimon/{clusters/21 msgr-failures/many msgr/simple no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} -- /tmp/teuthology-worker.xumwt4tx.tmp
2020-11-19T21:42:42.687 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-11-19T21:42:42.815 INFO:teuthology.run:Config:
  archive_path: /home/teuthworker/archive/yuriw-2020-11-19_17:21:50-rados-wip-yuri4-testing-2020-11-19-0746-nautilus-distro-basic-smithi/5637915
  branch: wip-yuri4-testing-2020-11-19-0746-nautilus
  description: rados/multimon/{clusters/21 msgr-failures/many msgr/simple no_pools objectstore/bluestore-comp-snappy
    rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews}
  email: ceph-qa@ceph.io
  first_in_suite: false
  job_id: '5637915'
  kernel:
    kdb: true
    sha1: distro
  last_in_suite: false
  machine_type: smithi
  name: yuriw-2020-11-19_17:21:50-rados-wip-yuri4-testing-2020-11-19-0746-nautilus-distro-basic-smithi
  nuke-on-error: true
  openstack:
  - volumes:
      count: 1
      size: 10
  os_type: centos
  os_version: '7.8'
  overrides:
    admin_socket:
      branch: wip-yuri4-testing-2020-11-19-0746-nautilus
    ceph:
      conf:
        global:
          mon mgr beacon grace: 90
          ms bind msgr2: false
          ms inject socket failures: 1000
          ms type: simple
        mgr:
          debug mgr: 20
          debug ms: 1
        mon:
          debug mon: 20
          debug ms: 1
          debug paxos: 20
        osd:
          bluestore block size: 96636764160
          bluestore compression algorithm: snappy
          bluestore compression mode: aggressive
          bluestore fsck on mount: true
          debug bluefs: 20
          debug bluestore: 20
          debug filestore: 20
          debug journal: 20
          debug ms: 20
          debug osd: 25
          debug rocksdb: 10
          mon osd backfillfull_ratio: 0.85
          mon osd full ratio: 0.9
          mon osd nearfull ratio: 0.8
          osd debug verify cached snaps: true
          osd debug verify missing on start: true
          osd failsafe full ratio: 0.95
          osd objectstore: bluestore
          osd op queue: debug_random
          osd op queue cut off: debug_random
      create_rbd_pool: false
      fs: xfs
      log-ignorelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      log-whitelist:
      - \(MDS_ALL_DOWN\)
      - \(MDS_UP_LESS_THAN_MAX\)
      - \(OSD_SLOW_PING_TIME
      mon_bind_msgr2: false
      sha1: a7ab790d3358b8b54c154754116c26711018b37e
    ceph-deploy:
      conf:
        client:
          log file: /var/log/ceph/ceph-$name.$pid.log
        mon:
          osd default pool size: 2
    install:
      ceph:
        sha1: a7ab790d3358b8b54c154754116c26711018b37e
    thrashosds:
      bdev_inject_crash: 2
      bdev_inject_crash_probability: 0.5
    workunit:
      branch: wip-yuri4-testing-2020-11-19-0746-nautilus
      sha1: a7ab790d3358b8b54c154754116c26711018b37e
  owner: scheduled_yuriw@teuthology
  priority: 80
  repo: https://github.com/ceph/ceph-ci.git
  roles:
  - - mon.a
    - mon.d
    - mon.g
    - mon.j
    - mon.m
    - mon.p
    - mon.s
  - - mon.b
    - mon.e
    - mon.h
    - mon.k
    - mon.n
    - mon.q
    - mon.t
    - mgr.x
  - - mon.c
    - mon.f
    - mon.i
    - mon.l
    - mon.o
    - mon.r
    - mon.u
  sha1: a7ab790d3358b8b54c154754116c26711018b37e
  sleep_before_teardown: 0
  suite: rados
  suite_branch: wip-yuri4-testing-2020-11-19-0746-nautilus
  suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2020-11-19-0746-nautilus/qa
  suite_relpath: qa
  suite_repo: https://github.com/ceph/ceph-ci.git
  suite_sha1: a7ab790d3358b8b54c154754116c26711018b37e
  tasks:
  - install: null
  - exec:
      mon.b:
      - sudo systemctl stop chronyd.service || true
      - sudo systemctl stop systemd-timesync.service || true
      - sudo systemctl stop ntpd.service || true
      - sudo systemctl stop ntp.service || true
      - date -u -s @$(expr $(date -u +%s) + 2)
  - ceph:
      log-whitelist:
      - .*clock.*skew.*
      - clocks not synchronized
      - overall HEALTH_
      - \(MON_CLOCK_SKEW\)
      - \(MGR_DOWN\)
      - \(MON_DOWN\)
      - \(PG_
      - \(SLOW_OPS\)
      - No standby daemons available
      - slow request
      wait-for-healthy: false
  - mon_clock_skew_check:
      expect-skew: true
  teuthology_branch: master
  timestamp: 2020-11-19_17:21:50
  tube: smithi
  user: yuriw
  verbose: true
  worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.6655
2020-11-19T21:42:42.815 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2020-11-19-0746-nautilus/qa; will attempt to use it
2020-11-19T21:42:42.816 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2020-11-19-0746-nautilus/qa/tasks
2020-11-19T21:42:42.817 INFO:teuthology.run_tasks:Running task internal.check_packages...
2020-11-19T21:42:42.880 INFO:teuthology.task.internal:Checking packages...
2020-11-19T21:42:42.980 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'a7ab790d3358b8b54c154754116c26711018b37e'
2020-11-19T21:42:42.980 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2020-11-19T21:42:42.981 INFO:teuthology.packaging:ref: None
2020-11-19T21:42:42.982 INFO:teuthology.packaging:tag: None
2020-11-19T21:42:42.982 INFO:teuthology.packaging:branch: wip-yuri4-testing-2020-11-19-0746-nautilus
2020-11-19T21:42:42.982 INFO:teuthology.packaging:sha1: a7ab790d3358b8b54c154754116c26711018b37e
2020-11-19T21:42:42.983 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-yuri4-testing-2020-11-19-0746-nautilus
2020-11-19T21:42:43.454 INFO:teuthology.task.internal:Found packages for ceph version 14.2.14-4.ga7ab790d
2020-11-19T21:42:43.455 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep...
2020-11-19T21:42:43.470 INFO:teuthology.task.internal:no buildpackages task found
2020-11-19T21:42:43.471 INFO:teuthology.run_tasks:Running task internal.lock_machines...
2020-11-19T21:42:43.485 INFO:teuthology.task.internal.lock_machines:Locking machines...
2020-11-19T21:42:43.486 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-11-19T21:42:43.539 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:42:53.563 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:43:03.579 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:43:13.625 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:43:23.659 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:43:33.685 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:43:43.709 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:43:53.728 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:44:03.743 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:44:13.833 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:44:23.855 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:44:33.979 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:44:43.994 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 1)...
2020-11-19T21:44:54.046 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:45:04.067 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:45:14.089 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-11-19T21:45:24.107 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:45:34.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:45:44.147 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:45:54.163 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:46:04.183 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:46:14.204 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:46:24.231 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:46:34.249 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:46:44.269 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-11-19T21:46:54.296 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:47:04.315 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:47:14.329 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:47:24.354 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:47:34.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:47:44.390 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:47:54.413 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:48:04.427 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:48:14.446 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 2)...
2020-11-19T21:48:24.459 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:48:34.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 3, have 0)...
2020-11-19T21:48:44.516 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_yuriw@teuthology', 'count': 3, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/yuriw-2020-11-19_17:21:50-rados-wip-yuri4-testing-2020-11-19-0746-nautilus-distro-basic-smithi/5637915'}
2020-11-19T21:48:44.563 DEBUG:teuthology.lock.ops:locked ubuntu@smithi101.front.sepia.ceph.com, ubuntu@smithi107.front.sepia.ceph.com, ubuntu@smithi158.front.sepia.ceph.com
2020-11-19T21:48:45.068 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f60c9671da0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f60c7fb9d90>, signals=[15])
2020-11-19T21:48:45.759 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of centos 7.8
2020-11-19T21:48:45.827 INFO:teuthology.provision.fog.smithi101:Scheduling deploy of centos 7.8
2020-11-19T21:48:45.847 INFO:teuthology.provision.fog.smithi158:Scheduling deploy of centos 7.8
2020-11-19T21:48:46.671 INFO:teuthology.orchestra.console:Power off smithi101
2020-11-19T21:48:46.675 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2020-11-19T21:48:46.774 INFO:teuthology.orchestra.console:Power off smithi107
2020-11-19T21:48:46.774 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2020-11-19T21:48:46.793 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:46.842 INFO:teuthology.orchestra.console:Power off smithi158
2020-11-19T21:48:46.843 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2020-11-19T21:48:46.860 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:46.886 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:50.898 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:50.925 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:50.951 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:55.246 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:55.260 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:55.342 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:55.553 INFO:teuthology.orchestra.console:Power off for smithi101 completed
2020-11-19T21:48:55.554 INFO:teuthology.orchestra.console:Power off for smithi107 completed
2020-11-19T21:48:55.652 INFO:teuthology.orchestra.console:Power off for smithi158 completed
2020-11-19T21:48:55.655 INFO:teuthology.orchestra.console:Power on smithi101
2020-11-19T21:48:55.655 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-11-19T21:48:55.776 INFO:teuthology.orchestra.console:Power on smithi107
2020-11-19T21:48:55.776 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-11-19T21:48:55.788 INFO:teuthology.orchestra.console:Power on smithi158
2020-11-19T21:48:55.789 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-11-19T21:48:55.818 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:55.827 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:55.845 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:59.846 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:59.909 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:48:59.928 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:49:00.134 INFO:teuthology.orchestra.console:Power on for smithi101 completed
2020-11-19T21:49:00.142 INFO:teuthology.orchestra.console:Power on for smithi158 completed
2020-11-19T21:49:00.235 INFO:teuthology.provision.fog.smithi101:Waiting for deploy to finish
2020-11-19T21:49:00.243 INFO:teuthology.provision.fog.smithi158:Waiting for deploy to finish
2020-11-19T21:49:03.980 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-11-19T21:49:04.287 INFO:teuthology.orchestra.console:Power on for smithi107 completed
2020-11-19T21:49:04.394 INFO:teuthology.provision.fog.smithi107:Waiting for deploy to finish
2020-11-19T21:51:31.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:51:31.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:51:35.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:51:56.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:51:56.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:51:59.884 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:05.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:05.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:08.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:14.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:14.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:17.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:23.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:23.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:27.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:32.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:32.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:36.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:52:41.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:53:09.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:53:10.006 INFO:teuthology.orchestra.run:Running command with timeout 600
2020-11-19T21:53:10.008 INFO:teuthology.orchestra.run.smithi101:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2020-11-19T21:53:13.549 INFO:teuthology.orchestra.run:Running command with timeout 600
2020-11-19T21:53:13.549 INFO:teuthology.orchestra.run.smithi158:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2020-11-19T21:53:13.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:53:15.226 INFO:teuthology.orchestra.run:Running command with timeout 600
2020-11-19T21:53:15.227 INFO:teuthology.orchestra.run.smithi107:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2020-11-19T21:53:25.410 INFO:teuthology.provision.fog.smithi101:Node is ready
2020-11-19T21:53:25.410 INFO:teuthology.orchestra.run.smithi101:> hostname
2020-11-19T21:53:25.530 INFO:teuthology.orchestra.run.smithi101.stdout:smithi061
2020-11-19T21:53:25.531 INFO:teuthology.orchestra.run.smithi101:> grep smithi061 /etc/hosts
2020-11-19T21:53:25.614 INFO:teuthology.orchestra.run.smithi101.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061
2020-11-19T21:53:25.615 INFO:teuthology.orchestra.run.smithi101:> sudo hostname smithi101
2020-11-19T21:53:25.800 INFO:teuthology.orchestra.run.smithi101:> sudo sed -i -e 's/smithi061/smithi101/g' /etc/hosts
2020-11-19T21:53:25.895 INFO:teuthology.orchestra.run.smithi101:> sudo sed -i -e 's/172.21.15.61/172.21.15.101/g' /etc/hosts
2020-11-19T21:53:25.977 INFO:teuthology.orchestra.run.smithi101:> sudo sed -i -e 's/smithi061/smithi101/g' /etc/hostname
2020-11-19T21:53:26.048 INFO:teuthology.orchestra.run.smithi101:> sudo hostname smithi101
2020-11-19T21:53:26.114 INFO:teuthology.provision.fog.smithi101:Deploy complete!
2020-11-19T21:53:26.115 DEBUG:teuthology.parallel:result is None
2020-11-19T21:53:35.525 INFO:teuthology.provision.fog.smithi158:Node is ready
2020-11-19T21:53:35.526 INFO:teuthology.orchestra.run.smithi158:> hostname
2020-11-19T21:53:35.640 INFO:teuthology.orchestra.run.smithi158.stdout:smithi158.front.sepia.ceph.com
2020-11-19T21:53:35.641 INFO:teuthology.orchestra.run.smithi158:> grep smithi158.front.sepia.ceph.com /etc/hosts
2020-11-19T21:53:35.707 INFO:teuthology.orchestra.run.smithi158.stdout:172.21.15.158 smithi158.front.sepia.ceph.com smithi158
2020-11-19T21:53:35.707 INFO:teuthology.orchestra.run.smithi158:> sudo hostname smithi158
2020-11-19T21:53:35.844 INFO:teuthology.orchestra.run.smithi158:> sudo sed -i -e 's/smithi158.front.sepia.ceph.com/smithi158/g' /etc/hosts
2020-11-19T21:53:35.929 INFO:teuthology.orchestra.run.smithi158:> sudo sed -i -e 's/172.21.15.158/172.21.15.158/g' /etc/hosts
2020-11-19T21:53:36.004 INFO:teuthology.orchestra.run.smithi158:> sudo sed -i -e 's/smithi158.front.sepia.ceph.com/smithi158/g' /etc/hostname
2020-11-19T21:53:36.073 INFO:teuthology.orchestra.run.smithi158:> sudo hostname smithi158
2020-11-19T21:53:36.133 INFO:teuthology.provision.fog.smithi158:Deploy complete!
2020-11-19T21:53:36.133 DEBUG:teuthology.parallel:result is None
2020-11-19T21:53:36.767 INFO:teuthology.provision.fog.smithi107:Node is ready
2020-11-19T21:53:36.767 INFO:teuthology.orchestra.run.smithi107:> hostname
2020-11-19T21:53:36.894 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107.front.sepia.ceph.com
2020-11-19T21:53:36.895 INFO:teuthology.orchestra.run.smithi107:> grep smithi107.front.sepia.ceph.com /etc/hosts
2020-11-19T21:53:36.972 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107 smithi107.front.sepia.ceph.com smithi107
2020-11-19T21:53:36.972 INFO:teuthology.orchestra.run.smithi107:> sudo hostname smithi107
2020-11-19T21:53:37.139 INFO:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hosts
2020-11-19T21:53:37.255 INFO:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/172.21.15.107/172.21.15.107/g' /etc/hosts
2020-11-19T21:53:37.360 INFO:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hostname
2020-11-19T21:53:37.448 INFO:teuthology.orchestra.run.smithi107:> sudo hostname smithi107
2020-11-19T21:53:37.526 INFO:teuthology.provision.fog.smithi107:Deploy complete!
2020-11-19T21:53:37.527 DEBUG:teuthology.parallel:result is None
2020-11-19T21:53:37.747 INFO:teuthology.lock.ops:Checking smithi101.front.sepia.ceph.com
2020-11-19T21:53:37.748 INFO:teuthology.lock.ops:New key found. Updating...
2020-11-19T21:53:37.767 INFO:teuthology.lock.ops:Checking smithi107.front.sepia.ceph.com
2020-11-19T21:53:37.767 INFO:teuthology.lock.ops:New key found. Updating...
2020-11-19T21:53:37.783 INFO:teuthology.lock.ops:Checking smithi158.front.sepia.ceph.com
2020-11-19T21:53:37.783 INFO:teuthology.lock.ops:New key found. Updating...
2020-11-19T21:53:37.798 INFO:teuthology.lock.ops:Updating [smithi101.front.sepia.ceph.com]: set os type and version on server
2020-11-19T21:53:37.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-11-19T21:53:37.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:53:37.897 INFO:teuthology.orchestra.run.smithi101:> true
2020-11-19T21:53:38.270 INFO:teuthology.orchestra.run.smithi101:> uname -m
2020-11-19T21:53:38.336 INFO:teuthology.orchestra.run.smithi101.stdout:x86_64
2020-11-19T21:53:38.336 INFO:teuthology.orchestra.run.smithi101:> cat /etc/os-release
2020-11-19T21:53:38.407 INFO:teuthology.orchestra.run.smithi101.stdout:NAME="CentOS Linux"
2020-11-19T21:53:38.408 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION="7 (Core)"
2020-11-19T21:53:38.408 INFO:teuthology.orchestra.run.smithi101.stdout:ID="centos"
2020-11-19T21:53:38.408 INFO:teuthology.orchestra.run.smithi101.stdout:ID_LIKE="rhel fedora"
2020-11-19T21:53:38.408 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION_ID="7"
2020-11-19T21:53:38.409 INFO:teuthology.orchestra.run.smithi101.stdout:PRETTY_NAME="CentOS Linux 7 (Core)"
2020-11-19T21:53:38.409 INFO:teuthology.orchestra.run.smithi101.stdout:ANSI_COLOR="0;31"
2020-11-19T21:53:38.409 INFO:teuthology.orchestra.run.smithi101.stdout:CPE_NAME="cpe:/o:centos:centos:7"
2020-11-19T21:53:38.409 INFO:teuthology.orchestra.run.smithi101.stdout:HOME_URL="https://www.centos.org/"
2020-11-19T21:53:38.409 INFO:teuthology.orchestra.run.smithi101.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2020-11-19T21:53:38.410 INFO:teuthology.orchestra.run.smithi101.stdout:
2020-11-19T21:53:38.410 INFO:teuthology.orchestra.run.smithi101.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7"
2020-11-19T21:53:38.410 INFO:teuthology.orchestra.run.smithi101.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7"
2020-11-19T21:53:38.410 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2020-11-19T21:53:38.410 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7"
2020-11-19T21:53:38.410 INFO:teuthology.orchestra.run.smithi101.stdout:
2020-11-19T21:53:38.411 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server
2020-11-19T21:53:38.427 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: set os type and version on server
2020-11-19T21:53:38.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-11-19T21:53:38.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:53:38.522 INFO:teuthology.orchestra.run.smithi107:> true
2020-11-19T21:53:38.909 INFO:teuthology.orchestra.run.smithi107:> uname -m
2020-11-19T21:53:38.979 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64
2020-11-19T21:53:38.980 INFO:teuthology.orchestra.run.smithi107:> cat /etc/os-release
2020-11-19T21:53:39.036 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Linux"
2020-11-19T21:53:39.036 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="7 (Core)"
2020-11-19T21:53:39.037 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos"
2020-11-19T21:53:39.037 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora"
2020-11-19T21:53:39.037 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="7"
2020-11-19T21:53:39.037 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Linux 7 (Core)"
2020-11-19T21:53:39.038 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31"
2020-11-19T21:53:39.038 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:7"
2020-11-19T21:53:39.038 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.centos.org/"
2020-11-19T21:53:39.038 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2020-11-19T21:53:39.038 INFO:teuthology.orchestra.run.smithi107.stdout:
2020-11-19T21:53:39.039 INFO:teuthology.orchestra.run.smithi107.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7"
2020-11-19T21:53:39.039 INFO:teuthology.orchestra.run.smithi107.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7"
2020-11-19T21:53:39.039 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2020-11-19T21:53:39.039 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7"
2020-11-19T21:53:39.039 INFO:teuthology.orchestra.run.smithi107.stdout:
2020-11-19T21:53:39.040 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server
2020-11-19T21:53:39.055 INFO:teuthology.lock.ops:Updating [smithi158.front.sepia.ceph.com]: set os type and version on server
2020-11-19T21:53:39.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-11-19T21:53:39.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-11-19T21:53:39.152 INFO:teuthology.orchestra.run.smithi158:> true
2020-11-19T21:53:39.563 INFO:teuthology.orchestra.run.smithi158:> uname -m
2020-11-19T21:53:39.649 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64
2020-11-19T21:53:39.650 INFO:teuthology.orchestra.run.smithi158:> cat /etc/os-release
2020-11-19T21:53:39.720 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="CentOS Linux"
2020-11-19T21:53:39.721 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="7 (Core)"
2020-11-19T21:53:39.721 INFO:teuthology.orchestra.run.smithi158.stdout:ID="centos"
2020-11-19T21:53:39.721 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="rhel fedora"
2020-11-19T21:53:39.722 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="7"
2020-11-19T21:53:39.722 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="CentOS Linux 7 (Core)"
2020-11-19T21:53:39.722 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31"
2020-11-19T21:53:39.723 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:centos:centos:7"
2020-11-19T21:53:39.723 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://www.centos.org/"
2020-11-19T21:53:39.723 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2020-11-19T21:53:39.723 INFO:teuthology.orchestra.run.smithi158.stdout:
2020-11-19T21:53:39.724 INFO:teuthology.orchestra.run.smithi158.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7"
2020-11-19T21:53:39.724 INFO:teuthology.orchestra.run.smithi158.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7"
2020-11-19T21:53:39.724 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2020-11-19T21:53:39.724 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7"
2020-11-19T21:53:39.725 INFO:teuthology.orchestra.run.smithi158.stdout:
2020-11-19T21:53:39.725 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server
2020-11-19T21:53:39.742 INFO:teuthology.task.internal.lock_machines:3 smithi machines locked this try, 3/3 locked so far
2020-11-19T21:53:39.792 INFO:teuthology.task.internal.lock_machines:Locked targets:
  smithi101.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIlT2kHVou9OBYBH7c3pHiv8wZQjQjEGeHKh0wNdHbq+/Hvc9lfvzSJEFMmSP9o+9ggA/YvGdWK4pyKiufNlk3FjlXxx9/Qa6hDQxop9aRjXLyMoJvvsWdII6qqwEqyuqzT0nvRpWQU297iYyNTHALxhKX+0ENoje6zai6yGX/qfrghEn1i821DP3dXPe+FMBFKRYg2ebUTM2/xuq0n/JlMBsAichYJHhUEaFwTYi2c489i9ZrkJFcMn8PW2KgTSc8xB7t6wZGPYu2W7vZJ/YbU8eDSCoviJFLfa6ViE63/ZpbH3tkPrre74yWEoXgDP2REGbLyI5c3NohK4tEqr6l
  smithi107.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUghKDPn/BQgY9vJ2D3vFphbMeqwbaih4cPw1g8sHxptk/nr6U23n//dwjf7sbr/iDgM9p6yBLPMcJGP68IsyLviDJtMQVCZ2rKvk/IseLXHEzn4OqRTMBow76+a00CTEFC4ZOziOoSS7zcLvHsOHm/I3+dHvWezGo8WeYFgHkgRfo5W13gghEVryQD5r8sLTbfd1ZqC/COuqMmCp4NRFgHrMLPAYULsjmS+a4T/RUmIGuHTkN3Oc9XvUhPiGyG6xUTXgPtOiJ0iMH+Shzwikqqc9FqZwdgnWeseUa8izsnRcrdrP0FJkQZqiyus4/eWtM5WvvTUhYAtRbQckxnwE9
  smithi158.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCupBYpNb6qCMVu58K75UDI4ToOrSmK5wAo8DP/V58CyTqUIzJvXub3Sak5gTRggP/NPs4QJAAyYs4kwrtXov+an2R4Uv5mRnjWhGYiWlw/7nAZ3QlUlnc8Tk8MbptvzHUEi56LUhlyaFSyqTgybc/pBb/NHnAwc4YVDIIT8SSrhxV4m1BQW9gpU8NveQcnIs782fE63/xD/htSath/Kx58XKQPjwHc5LvgCLoBkDoI8O5unrt+O73mJq7HVIAXs8xkvqlVfmmY5Fbv5P+G41nYJdK1b+8Pm8RXp6SMo4a9XS6TCKYjXsPeW64ci8rbrDbPkboJ4D0a9H43UsCnQoeF
2020-11-19T21:53:39.793 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/
2020-11-19T21:53:39.850 INFO:teuthology.run_tasks:Running task internal.save_config...