2019-10-11T12:31:32.620 INFO:root:teuthology version: 1.0.0-471e0e3 2019-10-11T12:31:32.621 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.b7jvas.tmp -- --verbose --lock --description rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} --name kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira --block --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira/4400389 2019-10-11T12:31:32.677 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-11T12:31:32.731 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira/4400389 branch: wip-kefu-testing-2019-10-09-1243 description: rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} email: null first_in_suite: false job_id: '4400389' kernel: kdb: true sha1: distro last_in_suite: false machine_type: mira name: kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-kefu-testing-2019-10-09-1243 ceph: conf: global: ms inject socket failures: 5000 osd_async_recovery_min_cost: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd max backfills: 3 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow requests sha1: 0dca79308d83afd02b99aa1aa8839d156215298c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: 0dca79308d83afd02b99aa1aa8839d156215298c thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2019-10-09-1243 sha1: 0dca79308d83afd02b99aa1aa8839d156215298c owner: scheduled_kchai@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 sha1: 0dca79308d83afd02b99aa1aa8839d156215298c suite: rados suite_branch: wip-kefu-testing-2019-10-09-1243 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-10-09-1243/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 0dca79308d83afd02b99aa1aa8839d156215298c tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology_branch: master tube: mira verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.mira.11874 2019-10-11T12:31:32.731 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-10-09-1243/qa; will attempt to use it 2019-10-11T12:31:32.731 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-kefu-testing-2019-10-09-1243/qa/tasks 2019-10-11T12:31:32.732 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-10-11T12:31:32.733 INFO:teuthology.task.internal:Checking packages... 2019-10-11T12:31:32.746 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '0dca79308d83afd02b99aa1aa8839d156215298c' 2019-10-11T12:31:32.746 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-10-11T12:31:32.746 INFO:teuthology.packaging:ref: None 2019-10-11T12:31:32.746 INFO:teuthology.packaging:tag: None 2019-10-11T12:31:32.746 INFO:teuthology.packaging:branch: wip-kefu-testing-2019-10-09-1243 2019-10-11T12:31:32.746 INFO:teuthology.packaging:sha1: 0dca79308d83afd02b99aa1aa8839d156215298c 2019-10-11T12:31:32.747 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-kefu-testing-2019-10-09-1243 2019-10-11T12:31:32.914 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-6009.g0dca793 2019-10-11T12:31:32.914 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-10-11T12:31:32.920 INFO:teuthology.task.internal:no buildpackages task found 2019-10-11T12:31:32.920 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-10-11T12:31:32.929 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-10-11T12:31:32.929 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-10-11T12:31:32.976 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:31:42.994 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:31:53.008 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:32:03.024 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:32:13.041 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:32:23.058 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:32:33.072 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:32:43.089 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:32:53.098 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:33:03.113 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:33:13.128 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:33:23.146 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:33:33.171 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:33:43.180 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:33:53.193 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:34:03.202 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:34:13.216 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:34:23.231 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:34:33.247 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:34:43.261 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:34:53.276 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:35:03.293 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:35:13.315 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:35:23.331 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:35:33.349 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:35:43.363 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:35:53.381 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:36:03.399 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:36:13.412 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:36:23.428 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:36:33.444 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:36:43.456 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:36:53.472 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:37:03.489 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:37:13.505 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:37:23.520 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:37:33.531 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:37:43.548 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:37:53.569 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:38:03.586 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:38:13.605 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:38:23.620 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:38:33.630 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:38:43.644 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:38:53.657 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:39:03.669 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:39:13.686 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:39:23.701 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:39:33.717 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:39:43.729 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:39:53.744 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:40:03.754 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:40:13.771 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:40:23.782 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:40:33.792 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:40:43.809 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:40:53.826 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:41:03.870 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:41:13.884 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:41:23.901 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:41:33.918 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:41:43.930 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:41:53.947 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:42:03.964 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:42:13.981 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:42:23.998 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:42:34.013 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:42:44.029 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:42:54.044 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:43:04.061 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:43:14.076 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:43:24.093 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:43:34.110 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:43:44.127 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:43:54.144 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:44:04.159 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:44:14.176 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:44:24.193 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:44:34.210 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:44:44.222 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:44:54.239 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:45:04.257 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:45:14.275 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:45:24.289 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:45:34.306 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:45:44.319 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:45:54.337 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:46:04.353 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:46:14.370 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:46:24.386 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:46:34.403 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:46:44.418 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:46:54.428 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:47:04.445 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:47:14.461 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:47:24.472 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:47:34.488 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:47:44.505 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:47:54.522 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:48:04.539 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:48:14.551 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:48:24.568 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:48:34.579 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:48:44.596 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:48:54.611 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:49:04.626 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:49:14.641 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:49:24.657 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:49:34.673 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:49:44.688 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:49:54.704 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:50:04.722 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:50:14.735 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:50:24.753 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:50:34.763 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:50:44.773 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:50:54.785 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:51:04.800 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:51:14.818 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:51:24.828 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:51:34.846 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:51:44.867 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:51:54.880 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:52:04.896 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:52:14.912 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:52:24.929 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:52:34.945 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:52:44.960 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:52:54.970 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:53:04.984 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:53:15.001 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:53:25.017 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:53:35.031 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:53:45.048 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:53:55.065 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:54:05.080 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:54:15.095 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:54:25.112 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:54:35.130 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:54:45.144 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:54:55.161 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:55:05.178 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:55:15.198 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:55:25.215 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:55:35.229 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:55:45.246 INFO:teuthology.task.internal.lock_machines:waiting for more mira machines to be free (need 2 + 4, have 2)... 2019-10-11T12:55:55.260 DEBUG:teuthology.lock.ops:lock_many request: {'count': 4, 'locked_by': 'scheduled_kchai@teuthology', 'description': '/home/teuthworker/archive/kchai-2019-10-11_00:28:24-rados-wip-kefu-testing-2019-10-09-1243-distro-basic-mira/4400389', 'machine_type': 'mira'} 2019-10-11T12:55:55.277 DEBUG:teuthology.lock.ops:locked ubuntu@mira117.front.sepia.ceph.com, ubuntu@mira082.front.sepia.ceph.com, ubuntu@mira018.front.sepia.ceph.com, ubuntu@mira110.front.sepia.ceph.com 2019-10-11T12:55:55.568 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-10-11T12:55:55.799 INFO:teuthology.provision.fog.mira117:Scheduling deploy of centos 7.6 2019-10-11T12:55:55.808 INFO:teuthology.provision.fog.mira110:Scheduling deploy of centos 7.6 2019-10-11T12:55:55.828 INFO:teuthology.provision.fog.mira018:Scheduling deploy of centos 7.6 2019-10-11T12:55:55.842 INFO:teuthology.provision.fog.mira082:Scheduling deploy of centos 7.6 2019-10-11T12:55:56.166 INFO:teuthology.orchestra.console:Power off mira110 2019-10-11T12:55:56.166 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-11T12:55:56.230 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:55:56.291 INFO:teuthology.orchestra.console:Power off mira117 2019-10-11T12:55:56.292 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-11T12:55:56.347 INFO:teuthology.orchestra.console:Power off mira082 2019-10-11T12:55:56.348 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-11T12:55:56.383 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:55:56.447 INFO:teuthology.orchestra.console:Power off mira018 2019-10-11T12:55:56.447 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-10-11T12:55:56.473 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:55:56.500 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:55:56.531 INFO:teuthology.orchestra.console:Power off for mira110 completed 2019-10-11T12:55:56.600 INFO:teuthology.orchestra.console:Power off for mira117 completed 2019-10-11T12:55:56.632 INFO:teuthology.orchestra.console:Power on mira110 2019-10-11T12:55:56.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-11T12:55:56.661 INFO:teuthology.orchestra.console:Power off for mira082 completed 2019-10-11T12:55:56.661 INFO:teuthology.orchestra.console:Power off for mira018 completed 2019-10-11T12:55:56.675 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:55:56.703 INFO:teuthology.orchestra.console:Power on mira117 2019-10-11T12:55:56.703 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-11T12:55:56.743 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:55:56.770 INFO:teuthology.orchestra.console:Power on mira082 2019-10-11T12:55:56.770 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-11T12:55:56.799 INFO:teuthology.orchestra.console:Power on mira018 2019-10-11T12:55:56.799 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-10-11T12:55:56.826 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:55:56.858 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:00.734 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:00.858 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:00.894 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:00.922 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:04.889 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:04.989 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:05.023 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:05.052 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H mira018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-10-11T12:56:05.122 INFO:teuthology.orchestra.console:Power on for mira110 completed 2019-10-11T12:56:05.217 INFO:teuthology.orchestra.console:Power on for mira117 completed 2019-10-11T12:56:05.223 INFO:teuthology.provision.fog.mira110:Waiting for deploy to finish 2019-10-11T12:56:05.254 INFO:teuthology.orchestra.console:Power on for mira082 completed 2019-10-11T12:56:05.288 INFO:teuthology.orchestra.console:Power on for mira018 completed 2019-10-11T12:56:05.318 INFO:teuthology.provision.fog.mira117:Waiting for deploy to finish 2019-10-11T12:56:05.355 INFO:teuthology.provision.fog.mira082:Waiting for deploy to finish 2019-10-11T12:56:05.389 INFO:teuthology.provision.fog.mira018:Waiting for deploy to finish 2019-10-11T12:58:37.185 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira082.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:58:37.219 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira018.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:58:37.326 ERROR:teuthology.orchestra.connection:Error connecting to mira018.front.sepia.ceph.com Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 104, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 437, in connect passphrase, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 749, in _auth raise saved_exception AuthenticationException: Authentication failed. 2019-10-11T12:58:46.249 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira082.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:58:52.294 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira117.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:58:52.304 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira110.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:58:55.309 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira082.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:01.353 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira117.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:01.385 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira110.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:04.362 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira082.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:10.413 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira117.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:10.446 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira110.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:13.422 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira082.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:19.466 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira117.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:19.497 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira110.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:22.473 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira082.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:28.521 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira117.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:28.553 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'mira110.front.sepia.ceph.com', 'timeout': 60} 2019-10-11T12:59:29.083 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2019-10-11T12:59:29.083 DEBUG:teuthology.task.console_log:Killing console logger for mira110 2019-10-11T12:59:29.084 DEBUG:teuthology.task.console_log:Killing console logger for mira117 2019-10-11T12:59:29.084 DEBUG:teuthology.task.console_log:Killing console logger for mira082 2019-10-11T12:59:29.084 DEBUG:teuthology.task.console_log:Killing console logger for mira018 2019-10-11T12:59:29.085 DEBUG:teuthology.exit:Finished running handlers