2020-08-31T19:09:51.184 INFO:root:teuthology version: 1.0.0-1d15e8d 2020-08-31T19:09:51.187 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --name pdonnell-2020-08-31_18:14:29-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi --block --archive /home/teuthworker/archive/pdonnell-2020-08-31_18:14:29-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396020 --verbose --lock --description kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} -- /tmp/teuthology-worker.oq8dfe55.tmp 2020-08-31T19:09:51.228 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-08-31T19:09:51.290 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2020-08-31_18:14:29-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396020 branch: wip-pdonnell-testing-20200829.172552 description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} email: pdonnell@redhat.com first_in_suite: false job_id: '5396020' kernel: client: branch: testing kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2020-08-31_18:14:29-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-pdonnell-testing-20200829.172552 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd objectstore: filestore osd op complaint time: 180 osd shutdown pgref assert: true osd sloppy crc: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Metadata damage detected log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-pdonnell-testing-20200829.172552 sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf owner: scheduled_pdonnell@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.b - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.c - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - - client.1 - - client.2 - - client.3 sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf sleep_before_teardown: 0 suite: kcephfs suite_branch: wip-pdonnell-testing-20200829.172552 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200829.172552/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology_branch: rpm-test timestamp: 2020-08-31_18:14:29 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.29137 2020-08-31T19:09:51.290 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200829.172552/qa; will attempt to use it 2020-08-31T19:09:51.291 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200829.172552/qa/tasks 2020-08-31T19:09:51.292 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-08-31T19:09:51.294 INFO:teuthology.task.internal:Checking packages... 2020-08-31T19:09:51.317 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf' 2020-08-31T19:09:51.318 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-08-31T19:09:51.318 INFO:teuthology.packaging:ref: None 2020-08-31T19:09:51.319 INFO:teuthology.packaging:tag: None 2020-08-31T19:09:51.319 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20200829.172552 2020-08-31T19:09:51.319 INFO:teuthology.packaging:sha1: c5af0d54b89ffb9f4c441f7a4e930a89dee8fecf 2020-08-31T19:09:51.320 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20200829.172552 2020-08-31T19:09:51.492 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-4885.gc5af0d5 2020-08-31T19:09:51.493 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-08-31T19:09:51.503 INFO:teuthology.task.internal:no buildpackages task found 2020-08-31T19:09:51.503 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-08-31T19:09:51.516 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-08-31T19:09:51.517 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-08-31T19:09:51.592 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:10:01.609 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:10:11.630 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:10:21.657 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:10:31.677 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:10:41.698 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:10:51.719 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:11:01.746 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:11:11.789 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:11:21.809 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:11:31.829 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:11:41.849 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 5)... 2020-08-31T19:11:51.867 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:12:01.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:12:11.909 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 4)... 2020-08-31T19:12:21.930 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:12:31.953 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:12:41.975 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:12:51.993 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:13:02.007 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:13:12.025 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:13:22.043 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:13:32.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:13:42.089 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:13:52.129 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:14:02.153 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:14:12.179 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:14:22.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:14:32.220 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:14:42.241 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:14:52.261 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:15:02.284 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:15:12.308 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:15:22.328 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:15:32.356 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:15:42.377 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:15:52.411 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:16:02.433 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:16:12.472 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:16:22.498 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:16:32.520 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:16:42.551 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:16:52.570 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:17:02.607 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:17:12.624 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:17:22.672 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:17:32.689 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:17:42.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:17:52.731 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:18:02.753 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:18:12.773 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:18:22.803 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:18:32.816 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:18:42.837 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:18:52.862 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:19:02.885 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:19:12.905 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:19:22.928 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:19:32.949 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:19:42.969 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:19:53.007 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:20:03.027 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:20:13.047 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:20:23.070 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:20:33.090 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 4)... 2020-08-31T19:20:43.112 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 3)... 2020-08-31T19:20:53.134 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:21:03.152 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:21:13.173 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:21:23.207 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:21:33.230 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:21:43.249 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 3)... 2020-08-31T19:21:53.268 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:22:03.291 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:22:13.315 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:22:23.346 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:22:33.366 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:22:43.392 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:22:53.420 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:23:03.446 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:23:13.466 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:23:23.489 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:23:33.514 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:23:43.531 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:23:53.543 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:24:03.558 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:24:13.576 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:24:23.599 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:24:33.612 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:24:43.633 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:24:53.652 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:25:03.672 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:25:13.692 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:25:23.707 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:25:33.731 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:25:43.749 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:25:53.777 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:26:03.801 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:26:13.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:26:23.848 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:26:33.875 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:26:43.893 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:26:53.920 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:27:03.941 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:27:13.963 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:27:23.983 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:27:34.004 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:27:44.042 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:27:54.066 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:28:04.097 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:28:14.138 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 5)... 2020-08-31T19:28:24.157 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:28:34.179 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 4)... 2020-08-31T19:28:44.204 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:28:54.224 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:29:04.249 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:29:14.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:29:24.300 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:29:34.328 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:29:44.349 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:29:54.377 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:30:04.415 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:30:14.442 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:30:24.488 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:30:34.513 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:30:44.560 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:30:54.614 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:31:04.635 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:31:14.658 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:31:24.686 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:31:34.707 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:31:44.735 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:31:54.765 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:32:04.785 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:32:14.815 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:32:24.830 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:32:34.841 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:32:44.863 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:32:54.887 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:33:04.907 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:33:14.926 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:33:24.947 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:33:34.969 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:33:44.992 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:33:55.012 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:34:05.033 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:34:15.053 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:34:25.077 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:34:35.096 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:34:45.124 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:34:55.146 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:35:05.180 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 4)... 2020-08-31T19:35:15.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:35:25.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:35:35.249 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:35:45.268 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:35:55.290 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:36:05.307 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:36:15.324 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:36:25.352 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:36:35.375 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:36:45.388 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:36:55.406 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:37:05.425 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:37:15.443 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 3)... 2020-08-31T19:37:25.465 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 3)... 2020-08-31T19:37:35.482 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 3)... 2020-08-31T19:37:45.502 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 3)... 2020-08-31T19:37:55.521 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:38:05.540 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:38:15.562 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:38:25.580 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:38:35.602 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:38:45.622 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:38:55.646 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:39:05.657 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 2)... 2020-08-31T19:39:15.674 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:39:25.692 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:39:35.718 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:39:45.735 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:39:55.752 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:40:05.776 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:40:15.798 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:40:25.815 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:40:35.837 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 1)... 2020-08-31T19:40:45.856 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:40:55.872 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:41:05.894 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:41:15.920 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:41:25.939 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:41:35.960 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:41:45.980 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:41:56.001 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:42:06.023 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:42:16.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 0)... 2020-08-31T19:42:26.075 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 6, have 3)... 2020-08-31T19:42:36.100 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 6, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-08-31_18:14:29-kcephfs-wip-pdonnell-testing-20200829.172552-distro-basic-smithi/5396020'} 2020-08-31T19:42:36.131 DEBUG:teuthology.lock.ops:locked ubuntu@smithi201.front.sepia.ceph.com, ubuntu@smithi029.front.sepia.ceph.com, ubuntu@smithi181.front.sepia.ceph.com, ubuntu@smithi112.front.sepia.ceph.com, ubuntu@smithi150.front.sepia.ceph.com, ubuntu@smithi124.front.sepia.ceph.com 2020-08-31T19:42:36.410 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fad3515b048>, signals=[15]) 2020-08-31T19:42:36.649 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of centos 8.1 2020-08-31T19:42:36.672 INFO:teuthology.provision.fog.smithi124:Scheduling deploy of centos 8.1 2020-08-31T19:42:36.675 INFO:teuthology.provision.fog.smithi201:Scheduling deploy of centos 8.1 2020-08-31T19:42:36.677 INFO:teuthology.provision.fog.smithi181:Scheduling deploy of centos 8.1 2020-08-31T19:42:36.682 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of centos 8.1 2020-08-31T19:42:36.685 INFO:teuthology.provision.fog.smithi112:Scheduling deploy of centos 8.1 2020-08-31T19:42:37.038 INFO:teuthology.orchestra.console:Power off smithi112 2020-08-31T19:42:37.039 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-31T19:42:37.091 INFO:teuthology.orchestra.console:Power off smithi150 2020-08-31T19:42:37.092 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-31T19:42:37.105 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:37.133 INFO:teuthology.orchestra.console:Power off smithi201 2020-08-31T19:42:37.134 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-31T19:42:37.156 INFO:teuthology.orchestra.console:Power off smithi029 2020-08-31T19:42:37.157 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-31T19:42:37.189 INFO:teuthology.orchestra.console:Power off smithi181 2020-08-31T19:42:37.191 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-31T19:42:37.206 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:37.234 INFO:teuthology.orchestra.console:Power off smithi124 2020-08-31T19:42:37.236 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-31T19:42:37.248 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:37.260 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:37.279 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:37.291 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:41.250 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:41.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:41.316 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:41.328 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:41.342 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:41.362 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.378 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.418 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.432 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.446 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.465 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.479 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.597 INFO:teuthology.orchestra.console:Power off for smithi112 completed 2020-08-31T19:42:45.648 INFO:teuthology.orchestra.console:Power off for smithi029 completed 2020-08-31T19:42:45.668 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2020-08-31T19:42:45.680 INFO:teuthology.orchestra.console:Power off for smithi181 completed 2020-08-31T19:42:45.699 INFO:teuthology.orchestra.console:Power on smithi112 2020-08-31T19:42:45.700 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-31T19:42:45.717 INFO:teuthology.orchestra.console:Power off for smithi124 completed 2020-08-31T19:42:45.726 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.751 INFO:teuthology.orchestra.console:Power on smithi029 2020-08-31T19:42:45.752 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-31T19:42:45.772 INFO:teuthology.orchestra.console:Power on smithi150 2020-08-31T19:42:45.774 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-31T19:42:45.787 INFO:teuthology.orchestra.console:Power on smithi181 2020-08-31T19:42:45.788 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-31T19:42:45.804 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.817 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.833 INFO:teuthology.orchestra.console:Power on smithi124 2020-08-31T19:42:45.835 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-31T19:42:45.850 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:45.872 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:49.538 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:49.763 INFO:teuthology.orchestra.console:Power off for smithi201 completed 2020-08-31T19:42:49.804 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:49.871 INFO:teuthology.orchestra.console:Power on smithi201 2020-08-31T19:42:49.872 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-31T19:42:49.888 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:49.901 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:49.925 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi181.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:49.942 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:49.956 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:50.144 INFO:teuthology.orchestra.console:Power on for smithi181 completed 2020-08-31T19:42:50.246 INFO:teuthology.provision.fog.smithi181:Waiting for deploy to finish 2020-08-31T19:42:53.971 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:53.994 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:54.008 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:54.028 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:54.057 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:54.187 INFO:teuthology.orchestra.console:Power on for smithi112 completed 2020-08-31T19:42:54.223 INFO:teuthology.orchestra.console:Power on for smithi150 completed 2020-08-31T19:42:54.282 INFO:teuthology.orchestra.console:Power on for smithi124 completed 2020-08-31T19:42:54.298 INFO:teuthology.provision.fog.smithi112:Waiting for deploy to finish 2020-08-31T19:42:54.325 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish 2020-08-31T19:42:54.363 INFO:teuthology.orchestra.console:Power on for smithi029 completed 2020-08-31T19:42:54.391 INFO:teuthology.provision.fog.smithi124:Waiting for deploy to finish 2020-08-31T19:42:54.469 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish 2020-08-31T19:42:58.110 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi201.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-31T19:42:58.325 INFO:teuthology.orchestra.console:Power on for smithi201 completed 2020-08-31T19:42:58.427 INFO:teuthology.provision.fog.smithi201:Waiting for deploy to finish 2020-08-31T19:45:21.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:25.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:25.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:25.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:25.627 ERROR:teuthology.orchestra.connection:Error connecting to smithi150.front.sepia.ceph.com Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 446, in connect passphrase, File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 764, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 740, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 1580, in auth_publickey return self.auth_handler.wait_for_response(my_event) File "/home/teuthworker/src/git.ceph.com_git_teuthology_rpm-test/virtualenv/lib/python3.6/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response raise e paramiko.ssh_exception.AuthenticationException: Authentication failed. 2020-08-31T19:45:29.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:30.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:34.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:34.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:38.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:39.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:40.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:43.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:43.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:47.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:48.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:49.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:52.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:52.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:56.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:57.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:45:58.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:01.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:01.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:05.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:06.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:07.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:10.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:10.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:14.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:15.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:16.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:19.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:19.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:23.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi201.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:24.821 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:25.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-31T19:46:31.238 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2020-08-31T19:46:31.240 DEBUG:teuthology.task.console_log:Killing console logger for smithi201 2020-08-31T19:46:31.242 DEBUG:teuthology.task.console_log:Killing console logger for smithi029 2020-08-31T19:46:31.242 DEBUG:teuthology.task.console_log:Killing console logger for smithi181 2020-08-31T19:46:31.243 DEBUG:teuthology.task.console_log:Killing console logger for smithi112 2020-08-31T19:46:31.243 DEBUG:teuthology.task.console_log:Killing console logger for smithi150 2020-08-31T19:46:31.244 DEBUG:teuthology.task.console_log:Killing console logger for smithi124 2020-08-31T19:46:31.247 DEBUG:teuthology.exit:Finished running handlers