2020-01-16T00:15:50.933 INFO:root:teuthology version: 1.0.0-7a27738 2020-01-16T00:15:50.934 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.YW0g9R.tmp -- --verbose --lock --description rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/few.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_8.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} --name sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi --block --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671722 2020-01-16T00:15:50.966 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-01-16T00:15:51.022 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671722 branch: wip-sage-testing-2020-01-15-1209 description: rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/few.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_8.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} email: null first_in_suite: false job_id: '4671722' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: '8.0' overrides: admin_socket: branch: wip-sage-testing-2020-01-15-1209 ceph: conf: global: mon client directed command retry: 5 ms inject socket failures: 5000 osd_async_recovery_min_cost: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd heartbeat use min delay socket: true osd max backfills: 3 osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd sloppy crc: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow requests sha1: cb3c85e74500283e01cc592f7e2858e323186910 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: sha1: cb3c85e74500283e01cc592f7e2858e323186910 workunit: branch: wip-sage-testing-2020-01-15-1209 sha1: cb3c85e74500283e01cc592f7e2858e323186910 owner: scheduled_sage@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 sha1: cb3c85e74500283e01cc592f7e2858e323186910 suite: rados suite_branch: wip-sage-testing-2020-01-15-1209 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: cb3c85e74500283e01cc592f7e2858e323186910 tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.17167 2020-01-16T00:15:51.023 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa; will attempt to use it 2020-01-16T00:15:51.023 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks 2020-01-16T00:15:51.023 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-01-16T00:15:51.025 INFO:teuthology.task.internal:Checking packages... 2020-01-16T00:15:51.040 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'cb3c85e74500283e01cc592f7e2858e323186910' 2020-01-16T00:15:51.040 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-01-16T00:15:51.040 INFO:teuthology.packaging:ref: None 2020-01-16T00:15:51.040 INFO:teuthology.packaging:tag: None 2020-01-16T00:15:51.041 INFO:teuthology.packaging:branch: wip-sage-testing-2020-01-15-1209 2020-01-16T00:15:51.041 INFO:teuthology.packaging:sha1: cb3c85e74500283e01cc592f7e2858e323186910 2020-01-16T00:15:51.041 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-sage-testing-2020-01-15-1209 2020-01-16T00:15:51.193 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-9353.gcb3c85e 2020-01-16T00:15:51.193 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-01-16T00:15:51.207 INFO:teuthology.task.internal:no buildpackages task found 2020-01-16T00:15:51.207 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-01-16T00:15:51.253 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-01-16T00:15:51.253 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-01-16T00:15:51.303 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:16:01.318 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:16:11.338 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:16:21.355 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:16:31.371 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:16:41.386 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:16:51.402 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:17:01.420 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:17:11.437 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:17:21.451 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:17:31.460 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:17:41.476 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:17:51.491 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:18:01.501 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:18:11.519 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:18:21.536 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:18:31.554 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:18:41.572 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-01-16T00:18:51.586 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:19:01.603 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:19:11.620 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:19:21.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:19:31.654 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:19:41.672 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:19:51.687 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:20:01.702 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:20:11.719 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:20:21.737 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:20:31.754 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:20:41.770 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:20:51.788 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:21:01.806 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:21:11.823 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:21:21.844 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:21:31.861 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:21:41.881 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:21:51.899 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:22:01.917 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:22:11.934 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:22:21.950 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:22:31.967 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:22:41.982 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:22:51.999 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:23:02.021 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:23:12.031 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:23:22.049 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:23:32.066 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:23:42.082 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:23:52.098 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:24:02.115 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:24:12.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:24:22.144 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:24:32.160 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:24:42.188 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:24:52.202 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:25:02.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:25:12.228 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 1)... 2020-01-16T00:25:22.246 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:25:32.264 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:25:42.311 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:25:52.325 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:26:02.335 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:26:12.351 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:26:22.361 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:26:32.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:26:42.399 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:26:52.412 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:27:02.430 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:27:12.451 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:27:22.467 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:27:32.479 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:27:42.494 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:27:52.511 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:28:02.528 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:28:12.541 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:28:22.557 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:28:32.573 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:28:42.598 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:28:52.613 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:29:02.625 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-01-16T00:29:12.641 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:29:22.657 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:29:32.677 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:29:42.705 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:29:52.721 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:30:02.748 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)... 2020-01-16T00:30:12.769 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:30:22.787 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:30:32.805 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-01-16T00:30:42.814 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-01-16T00:30:52.824 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-01-16T00:31:02.837 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 5)... 2020-01-16T00:31:12.861 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:31:22.877 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:31:32.897 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:31:42.913 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:31:52.970 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:32:02.989 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:32:13.006 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:32:23.021 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:32:33.039 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:32:43.056 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:32:53.069 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:33:03.082 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:33:13.098 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:33:23.116 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:33:33.133 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 0)... 2020-01-16T00:33:43.150 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:33:53.164 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:34:03.174 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:34:13.192 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:34:23.206 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:34:33.225 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:34:43.243 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:34:53.259 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:35:03.280 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:35:13.299 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:35:23.316 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:35:33.332 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:35:43.346 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:35:53.362 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:36:03.378 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:36:13.393 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:36:23.413 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:36:33.427 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:36:43.439 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:36:53.450 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:37:03.460 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:37:13.480 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:37:23.497 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:37:33.510 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:37:43.528 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:37:53.545 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 4)... 2020-01-16T00:38:03.570 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:38:13.587 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:38:23.603 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:38:33.622 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:38:43.639 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:38:53.654 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:39:03.670 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:39:13.689 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:39:23.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:39:33.729 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:39:43.745 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:39:53.764 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:40:03.782 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:40:13.802 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:40:23.819 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:40:33.835 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:40:43.850 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:40:53.868 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:41:03.885 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:41:13.898 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:41:23.916 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:41:33.925 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 2)... 2020-01-16T00:41:43.938 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 4, have 3)... 2020-01-16T00:41:53.956 DEBUG:teuthology.lock.ops:lock_many request: {'count': 4, 'locked_by': 'scheduled_sage@teuthology', 'description': '/home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671722', 'machine_type': 'smithi'} 2020-01-16T00:41:53.979 DEBUG:teuthology.lock.ops:locked ubuntu@smithi136.front.sepia.ceph.com, ubuntu@smithi162.front.sepia.ceph.com, ubuntu@smithi110.front.sepia.ceph.com, ubuntu@smithi080.front.sepia.ceph.com 2020-01-16T00:41:54.138 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-01-16T00:41:54.374 INFO:teuthology.provision.fog.smithi080:Scheduling deploy of centos 8.0 2020-01-16T00:41:54.376 INFO:teuthology.provision.fog.smithi162:Scheduling deploy of centos 8.0 2020-01-16T00:41:54.379 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of centos 8.0 2020-01-16T00:41:54.381 INFO:teuthology.provision.fog.smithi136:Scheduling deploy of centos 8.0 2020-01-16T00:41:54.763 INFO:teuthology.orchestra.console:Power off smithi080 2020-01-16T00:41:54.763 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-01-16T00:41:54.782 INFO:teuthology.orchestra.console:Power off smithi162 2020-01-16T00:41:54.783 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-01-16T00:41:54.818 INFO:teuthology.orchestra.console:Power off smithi110 2020-01-16T00:41:54.819 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-01-16T00:41:54.838 INFO:teuthology.orchestra.console:Power off smithi136 2020-01-16T00:41:54.838 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-01-16T00:41:54.846 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:41:54.853 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:41:54.861 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:41:54.870 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:41:54.979 INFO:teuthology.orchestra.console:Power off for smithi162 completed 2020-01-16T00:41:54.995 INFO:teuthology.orchestra.console:Power off for smithi136 completed 2020-01-16T00:41:55.081 INFO:teuthology.orchestra.console:Power on smithi162 2020-01-16T00:41:55.081 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-01-16T00:41:55.100 INFO:teuthology.orchestra.console:Power on smithi136 2020-01-16T00:41:55.100 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-01-16T00:41:55.109 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:41:55.121 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:41:58.882 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:41:58.894 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:41:59.150 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:41:59.160 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:02.998 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:03.009 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:03.264 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:03.278 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:03.478 INFO:teuthology.orchestra.console:Power on for smithi162 completed 2020-01-16T00:42:03.487 INFO:teuthology.orchestra.console:Power on for smithi136 completed 2020-01-16T00:42:03.580 INFO:teuthology.provision.fog.smithi162:Waiting for deploy to finish 2020-01-16T00:42:03.588 INFO:teuthology.provision.fog.smithi136:Waiting for deploy to finish 2020-01-16T00:42:07.114 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:07.123 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:07.324 INFO:teuthology.orchestra.console:Power off for smithi110 completed 2020-01-16T00:42:07.332 INFO:teuthology.orchestra.console:Power off for smithi080 completed 2020-01-16T00:42:07.425 INFO:teuthology.orchestra.console:Power on smithi110 2020-01-16T00:42:07.425 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-01-16T00:42:07.434 INFO:teuthology.orchestra.console:Power on smithi080 2020-01-16T00:42:07.434 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-01-16T00:42:07.446 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:07.457 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:11.470 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:11.490 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:11.698 INFO:teuthology.orchestra.console:Power on for smithi080 completed 2020-01-16T00:42:11.799 INFO:teuthology.provision.fog.smithi080:Waiting for deploy to finish 2020-01-16T00:42:15.586 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-16T00:42:15.795 INFO:teuthology.orchestra.console:Power on for smithi110 completed 2020-01-16T00:42:15.896 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish 2020-01-16T00:44:34.578 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi162.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:44:34.664 ERROR:teuthology.orchestra.connection:Error connecting to smithi162.front.sepia.ceph.com Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 446, in connect passphrase, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 764, in _auth raise saved_exception AuthenticationException: Authentication failed. 2020-01-16T00:44:49.640 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:44:57.916 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi080.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:44:58.711 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:45:01.990 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:45:07.767 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:45:11.062 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:45:16.823 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:45:20.119 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:45:25.883 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:45:29.175 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi110.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:45:34.939 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:45:35.966 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2020-01-16T00:45:35.967 DEBUG:teuthology.task.console_log:Killing console logger for smithi162 2020-01-16T00:45:35.967 DEBUG:teuthology.task.console_log:Killing console logger for smithi080 2020-01-16T00:45:35.968 DEBUG:teuthology.task.console_log:Killing console logger for smithi136 2020-01-16T00:45:35.968 DEBUG:teuthology.task.console_log:Killing console logger for smithi110 2020-01-16T00:45:35.968 DEBUG:teuthology.exit:Finished running handlers