2020-02-08T18:02:23.418 INFO:root:teuthology version: 1.0.0-ee19bca 2020-02-08T18:02:23.418 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.XDfiqn.tmp -- --verbose --lock --description rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/nautilus-v1only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_7.6.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/mapgap.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} --name sage-2020-02-08_17:26:32-rados-wip-sage2-testing-2020-02-08-0831-distro-basic-smithi --block --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2020-02-08_17:26:32-rados-wip-sage2-testing-2020-02-08-0831-distro-basic-smithi/4745670 2020-02-08T18:02:23.469 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-08T18:02:23.529 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2020-02-08_17:26:32-rados-wip-sage2-testing-2020-02-08-0831-distro-basic-smithi/4745670 branch: wip-sage2-testing-2020-02-08-0831 description: rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/nautilus-v1only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_7.6.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/mapgap.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} email: null first_in_suite: false job_id: '4745670' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2020-02-08_17:26:32-rados-wip-sage2-testing-2020-02-08-0831-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-sage2-testing-2020-02-08-0831 ceph: conf: global: mon client directed command retry: 5 ms bind msgr2: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 3 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd backoff on peering: true osd debug verify cached snaps: true osd debug verify missing on start: true osd map cache size: 1 osd max backfills: 6 osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow requests mon_bind_msgr2: false sha1: f366546002698478e7b1888bb3d9023bb85c3032 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: f366546002698478e7b1888bb3d9023bb85c3032 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-sage2-testing-2020-02-08-0831 sha1: f366546002698478e7b1888bb3d9023bb85c3032 owner: scheduled_sage@teuthology priority: 80 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 sha1: f366546002698478e7b1888bb3d9023bb85c3032 suite: rados suite_branch: wip-sage2-testing-2020-02-08-0831 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2020-02-08-0831/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: f366546002698478e7b1888bb3d9023bb85c3032 tasks: - install: branch: nautilus exclude_packages: - cephadm - ceph-mgr-cephadm - ceph-immutable-object-cache - python3-rados - python3-rgw - python3-rbd - python3-cephfs extra_packages: - python-rados - python-rgw - python-rbd - python-cephfs - cephadm: null - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - rados: clients: - client.2 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.11873 2020-02-08T18:02:23.530 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2020-02-08-0831/qa; will attempt to use it 2020-02-08T18:02:23.530 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2020-02-08-0831/qa/tasks 2020-02-08T18:02:23.531 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-02-08T18:02:23.532 INFO:teuthology.task.internal:Checking packages... 2020-02-08T18:02:23.547 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'f366546002698478e7b1888bb3d9023bb85c3032' 2020-02-08T18:02:23.547 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-02-08T18:02:23.547 INFO:teuthology.packaging:ref: None 2020-02-08T18:02:23.547 INFO:teuthology.packaging:tag: None 2020-02-08T18:02:23.547 INFO:teuthology.packaging:branch: wip-sage2-testing-2020-02-08-0831 2020-02-08T18:02:23.548 INFO:teuthology.packaging:sha1: f366546002698478e7b1888bb3d9023bb85c3032 2020-02-08T18:02:23.548 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-sage2-testing-2020-02-08-0831 2020-02-08T18:02:23.698 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-405.gf366546 2020-02-08T18:02:23.698 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-02-08T18:02:24.605 INFO:teuthology.task.internal:no buildpackages task found 2020-02-08T18:02:24.605 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-02-08T18:02:24.621 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-02-08T18:02:24.621 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-08T18:02:24.670 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:02:34.689 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:02:44.713 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:02:54.731 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:03:04.750 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:03:14.772 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:03:24.796 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:03:34.814 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:03:44.835 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:03:54.849 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:04:04.875 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:04:14.893 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:04:24.916 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:04:34.933 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:04:44.954 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:04:54.973 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:05:05.018 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:05:15.038 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:05:25.055 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:05:35.074 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:05:45.096 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:05:55.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:06:05.141 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:06:15.160 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:06:25.171 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:06:35.187 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:06:45.208 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:06:55.230 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:07:05.251 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:07:15.275 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:07:25.296 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:07:35.319 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:07:45.340 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:07:55.355 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:08:05.376 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:08:15.392 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:08:25.407 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:08:35.445 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:08:45.468 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:08:55.524 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:09:05.540 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:09:15.558 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:09:25.584 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:09:35.608 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:09:45.639 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:09:55.655 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:10:05.674 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:10:15.712 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:10:25.731 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:10:35.749 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:10:45.769 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:10:55.815 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:11:05.843 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:11:15.872 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:11:25.892 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:11:35.935 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:11:45.956 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:11:55.973 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:12:05.994 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:12:16.017 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:12:26.056 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:12:36.078 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:12:46.098 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:12:56.124 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:13:06.143 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:13:16.165 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:13:26.186 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:13:36.205 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:13:46.224 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:13:56.247 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:14:06.263 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:14:16.282 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:14:26.303 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:14:36.332 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:14:46.387 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:14:57.552 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:15:07.594 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:15:17.616 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:15:27.638 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:15:37.657 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:15:47.677 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:15:57.699 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:16:07.717 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:16:17.737 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:16:27.757 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:16:37.780 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:16:47.796 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:16:57.822 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:17:07.845 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:17:17.870 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:17:27.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:17:37.905 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:17:47.924 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 3)... 2020-02-08T18:17:57.947 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:18:07.964 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:18:17.978 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:18:27.995 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:18:38.010 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:18:48.027 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:18:58.045 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:19:08.065 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:19:18.082 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:19:28.100 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:19:38.123 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:19:48.143 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:19:58.162 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:20:08.177 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:20:18.198 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:20:28.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:20:38.237 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:20:48.253 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:20:58.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:21:08.288 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 4)... 2020-02-08T18:21:18.300 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:21:28.320 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:21:38.334 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:21:48.353 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:21:58.369 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:22:08.386 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:22:18.403 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:22:28.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 3, have 2)... 2020-02-08T18:22:38.434 DEBUG:teuthology.lock.ops:lock_many request: {'count': 3, 'locked_by': 'scheduled_sage@teuthology', 'description': '/home/teuthworker/archive/sage-2020-02-08_17:26:32-rados-wip-sage2-testing-2020-02-08-0831-distro-basic-smithi/4745670', 'machine_type': 'smithi'} 2020-02-08T18:22:38.455 DEBUG:teuthology.lock.ops:locked ubuntu@smithi136.front.sepia.ceph.com, ubuntu@smithi121.front.sepia.ceph.com, ubuntu@smithi052.front.sepia.ceph.com 2020-02-08T18:22:38.568 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-02-08T18:22:38.799 INFO:teuthology.provision.fog.smithi052:Scheduling deploy of centos 7.6 2020-02-08T18:22:38.802 INFO:teuthology.provision.fog.smithi136:Scheduling deploy of centos 7.6 2020-02-08T18:22:38.825 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of centos 7.6 2020-02-08T18:22:39.210 INFO:teuthology.orchestra.console:Power off smithi121 2020-02-08T18:22:39.210 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-02-08T18:22:39.234 INFO:teuthology.orchestra.console:Power off smithi136 2020-02-08T18:22:39.235 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-02-08T18:22:39.251 INFO:teuthology.orchestra.console:Power off smithi052 2020-02-08T18:22:39.251 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-02-08T18:22:39.261 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:39.273 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:39.283 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:43.294 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:43.311 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:43.319 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:47.410 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:47.422 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:47.436 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:47.638 INFO:teuthology.orchestra.console:Power off for smithi052 completed 2020-02-08T18:22:47.649 INFO:teuthology.orchestra.console:Power off for smithi136 completed 2020-02-08T18:22:47.742 INFO:teuthology.orchestra.console:Power on smithi052 2020-02-08T18:22:47.742 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-02-08T18:22:47.751 INFO:teuthology.orchestra.console:Power on smithi136 2020-02-08T18:22:47.752 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-02-08T18:22:47.767 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:47.784 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:51.526 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:51.738 INFO:teuthology.orchestra.console:Power off for smithi121 completed 2020-02-08T18:22:51.803 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:51.814 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:51.839 INFO:teuthology.orchestra.console:Power on smithi121 2020-02-08T18:22:51.840 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-02-08T18:22:51.863 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:55.890 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:55.915 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:55.926 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:22:56.126 INFO:teuthology.orchestra.console:Power on for smithi052 completed 2020-02-08T18:22:56.137 INFO:teuthology.orchestra.console:Power on for smithi136 completed 2020-02-08T18:22:56.227 INFO:teuthology.provision.fog.smithi052:Waiting for deploy to finish 2020-02-08T18:22:56.238 INFO:teuthology.provision.fog.smithi136:Waiting for deploy to finish 2020-02-08T18:23:00.001 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-08T18:23:00.211 INFO:teuthology.orchestra.console:Power on for smithi121 completed 2020-02-08T18:23:00.312 INFO:teuthology.provision.fog.smithi121:Waiting for deploy to finish 2020-02-08T18:25:12.377 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:25:21.439 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:25:30.525 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:25:39.610 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:25:42.627 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:25:46.653 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:25:48.663 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:25:51.703 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:25:55.738 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:25:57.719 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:00.760 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:04.795 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:06.780 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:09.827 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:13.851 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:15.831 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:18.910 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:22.907 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:24.887 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:27.991 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:31.959 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:33.949 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:37.054 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:41.019 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:43.031 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:46.138 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:50.071 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:52.087 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:55.194 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:26:59.130 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:27:01.145 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:27:25.658 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:27:34.718 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:27:35.020 INFO:teuthology.orchestra.run.smithi052:> hostname 2020-02-08T18:27:35.623 INFO:teuthology.orchestra.run.smithi052.stdout:smithi068 2020-02-08T18:27:35.624 INFO:teuthology.orchestra.run.smithi052:> grep smithi068 /etc/hosts 2020-02-08T18:27:35.699 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.68 smithi068.front.sepia.ceph.com smithi068 2020-02-08T18:27:35.700 INFO:teuthology.orchestra.run.smithi052:> sudo hostname smithi052 2020-02-08T18:27:35.852 INFO:teuthology.orchestra.run.smithi052:> sudo sed -i -e 's/smithi068/smithi052/g' /etc/hosts 2020-02-08T18:27:35.932 INFO:teuthology.orchestra.run.smithi052:> sudo sed -i -e 's/172.21.15.68/172.21.15.52/g' /etc/hosts 2020-02-08T18:27:35.992 INFO:teuthology.orchestra.run.smithi052:> sudo sed -i -e 's/smithi068/smithi052/g' /etc/hostname 2020-02-08T18:27:36.058 INFO:teuthology.orchestra.run.smithi052:> sudo hostname smithi052 2020-02-08T18:27:36.119 INFO:teuthology.provision.fog.smithi052:Deploy complete! 2020-02-08T18:27:36.121 DEBUG:teuthology.parallel:result is None 2020-02-08T18:28:01.215 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:28:01.404 INFO:teuthology.orchestra.run.smithi136:> hostname 2020-02-08T18:28:01.993 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136.front.sepia.ceph.com 2020-02-08T18:28:01.994 INFO:teuthology.orchestra.run.smithi136:> grep smithi136.front.sepia.ceph.com /etc/hosts 2020-02-08T18:28:02.073 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136 smithi136.front.sepia.ceph.com smithi136 2020-02-08T18:28:02.074 INFO:teuthology.orchestra.run.smithi136:> sudo hostname smithi136 2020-02-08T18:28:02.216 INFO:teuthology.orchestra.run.smithi136:> sudo sed -i -e 's/smithi136.front.sepia.ceph.com/smithi136/g' /etc/hosts 2020-02-08T18:28:02.317 INFO:teuthology.orchestra.run.smithi136:> sudo sed -i -e 's/172.21.15.136/172.21.15.136/g' /etc/hosts 2020-02-08T18:28:02.394 INFO:teuthology.orchestra.run.smithi136:> sudo sed -i -e 's/smithi136.front.sepia.ceph.com/smithi136/g' /etc/hostname 2020-02-08T18:28:02.459 INFO:teuthology.orchestra.run.smithi136:> sudo hostname smithi136 2020-02-08T18:28:02.529 INFO:teuthology.provision.fog.smithi136:Deploy complete! 2020-02-08T18:28:02.531 DEBUG:teuthology.parallel:result is None 2020-02-08T18:28:05.135 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:28:05.333 INFO:teuthology.orchestra.run.smithi121:> hostname 2020-02-08T18:28:05.876 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121 2020-02-08T18:28:05.876 INFO:teuthology.orchestra.run.smithi121:> grep smithi121 /etc/hosts 2020-02-08T18:28:05.959 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121 smithi121.front.sepia.ceph.com smithi121 2020-02-08T18:28:05.960 INFO:teuthology.orchestra.run.smithi121:> sudo hostname smithi121 2020-02-08T18:28:06.101 INFO:teuthology.orchestra.run.smithi121:> sudo sed -i -e 's/smithi121/smithi121/g' /etc/hosts 2020-02-08T18:28:06.190 INFO:teuthology.orchestra.run.smithi121:> sudo sed -i -e 's/172.21.15.121/172.21.15.121/g' /etc/hosts 2020-02-08T18:28:06.275 INFO:teuthology.orchestra.run.smithi121:> sudo sed -i -e 's/smithi121/smithi121/g' /etc/hostname 2020-02-08T18:28:06.350 INFO:teuthology.orchestra.run.smithi121:> sudo hostname smithi121 2020-02-08T18:28:06.413 INFO:teuthology.provision.fog.smithi121:Deploy complete! 2020-02-08T18:28:06.417 DEBUG:teuthology.parallel:result is None 2020-02-08T18:28:06.642 INFO:teuthology.lock.ops:Checking smithi136.front.sepia.ceph.com 2020-02-08T18:28:06.643 INFO:teuthology.lock.ops:New key found. Updating... 2020-02-08T18:28:06.676 INFO:teuthology.lock.ops:Checking smithi121.front.sepia.ceph.com 2020-02-08T18:28:06.676 INFO:teuthology.lock.ops:New key found. Updating... 2020-02-08T18:28:06.688 INFO:teuthology.lock.ops:Checking smithi052.front.sepia.ceph.com 2020-02-08T18:28:06.688 INFO:teuthology.lock.ops:New key found. Updating... 2020-02-08T18:28:06.706 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: set os type and version on server 2020-02-08T18:28:06.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-08T18:28:06.707 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:28:06.807 INFO:teuthology.orchestra.run.smithi136:> true 2020-02-08T18:28:07.178 INFO:teuthology.orchestra.run.smithi136:> uname -m 2020-02-08T18:28:07.258 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2020-02-08T18:28:07.258 INFO:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2020-02-08T18:28:07.335 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Linux" 2020-02-08T18:28:07.335 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="7 (Core)" 2020-02-08T18:28:07.335 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2020-02-08T18:28:07.335 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2020-02-08T18:28:07.336 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="7" 2020-02-08T18:28:07.336 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-02-08T18:28:07.336 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2020-02-08T18:28:07.336 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-02-08T18:28:07.336 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.centos.org/" 2020-02-08T18:28:07.336 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-02-08T18:28:07.336 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:07.337 INFO:teuthology.orchestra.run.smithi136.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-02-08T18:28:07.337 INFO:teuthology.orchestra.run.smithi136.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-02-08T18:28:07.337 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-02-08T18:28:07.337 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-02-08T18:28:07.337 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:07.338 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2020-02-08T18:28:07.370 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: set os type and version on server 2020-02-08T18:28:07.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-08T18:28:07.370 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:28:07.471 INFO:teuthology.orchestra.run.smithi121:> true 2020-02-08T18:28:07.852 INFO:teuthology.orchestra.run.smithi121:> uname -m 2020-02-08T18:28:07.916 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2020-02-08T18:28:07.916 INFO:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2020-02-08T18:28:08.000 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Linux" 2020-02-08T18:28:08.000 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="7 (Core)" 2020-02-08T18:28:08.000 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2020-02-08T18:28:08.000 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2020-02-08T18:28:08.001 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="7" 2020-02-08T18:28:08.001 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-02-08T18:28:08.001 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2020-02-08T18:28:08.001 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-02-08T18:28:08.001 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.centos.org/" 2020-02-08T18:28:08.002 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-02-08T18:28:08.002 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-02-08T18:28:08.002 INFO:teuthology.orchestra.run.smithi121.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-02-08T18:28:08.002 INFO:teuthology.orchestra.run.smithi121.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-02-08T18:28:08.003 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-02-08T18:28:08.003 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-02-08T18:28:08.003 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-02-08T18:28:08.003 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2020-02-08T18:28:08.030 INFO:teuthology.lock.ops:Updating [smithi052.front.sepia.ceph.com]: set os type and version on server 2020-02-08T18:28:08.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-08T18:28:08.031 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:28:08.134 INFO:teuthology.orchestra.run.smithi052:> true 2020-02-08T18:28:08.533 INFO:teuthology.orchestra.run.smithi052:> uname -m 2020-02-08T18:28:08.601 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2020-02-08T18:28:08.601 INFO:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2020-02-08T18:28:08.660 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Linux" 2020-02-08T18:28:08.661 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="7 (Core)" 2020-02-08T18:28:08.662 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2020-02-08T18:28:08.662 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2020-02-08T18:28:08.662 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="7" 2020-02-08T18:28:08.662 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-02-08T18:28:08.663 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2020-02-08T18:28:08.663 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-02-08T18:28:08.663 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.centos.org/" 2020-02-08T18:28:08.664 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-02-08T18:28:08.664 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:28:08.664 INFO:teuthology.orchestra.run.smithi052.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-02-08T18:28:08.665 INFO:teuthology.orchestra.run.smithi052.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-02-08T18:28:08.665 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-02-08T18:28:08.665 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-02-08T18:28:08.665 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:28:08.666 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2020-02-08T18:28:08.693 INFO:teuthology.task.internal.lock_machines:3 smithi machines locked this try, 3/3 locked so far 2020-02-08T18:28:08.750 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8wXMFQAiPFwiwcKs0ixt2NTITQ/FblmIX9RfoCAyhmjdVNCZJ13wOPFDlDWNLf3qD+imWoxZlELAcSQISIK1Ena6EwfPDzd2j8ZZOt6YfKnQ/cYF3JX2lqapnmcf0hx/S7fjcx9R0Jl96XQOWnAsdMIDHEykFTQNcLzH3kiSkxZ7p/MH10dmYJT2H0nUzSjmKHYptIGzzwPru+98hMjnJ4+U7HSYhol1ve4qX0q6qGGu0VasiRksLi/ePVIk5FjWxPvtKDPrVmAzxGKfVKwLErXkbsMY+MyiRjrJoj5+IIB2TBaELrF/FUTZ0l1ec1gzgybArLEmXjNHfjQMzTDcn smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMQtmHd3J4nMF1ACszQx9LAXMZ4bTbG35Qa4Y2O6Jfk+veyNmD6Dq9qQhRMtqQsOJyic4EJJ5wj2xNy5PFzFGyddN3q41Ak635rFCc3ellaPFJFlXsUMeQh1Xl9xkpG2coaEA/Z5RmwSVbHltPxsTa5tnnvtAStnJ8h7d4+YWfyW/vSNGxRckQIJYHkMRF1Z2n/MlYvwuvY9gdhemmQsPBoKojdiP9Cs0zbKYUsXfjCoDYj9vDzgzqbvGNZnIAGWLKPc98n4gTjPKBkH5orCLJxuiDs9i8beJ3a+tydrTWp3FRRWtWgNcGPglTqolSWFnWsgr+skaqfLlli7XPmY73 smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPnqE41CBonr14h9GkLWBvj9kFo4vGgORHel0tlL5sBQeLhK/FJHqiuuotXATCcuSpZxSw6ILxlWAFBryDgVuCCDFiiCkKDn6SjnNN8twJJLhnbGXpPabXbST9bucDuc9ylW+34WBY3sL+ruq4wybPKHoxHsnA6n91Q+AkkXe2nWBSA89sYv6SbINO8YP4J1Y+hffD8JoxO20VqIU/m/Eg8ERrK75qPE8GcPQwq69cezFqGFvkPr3po5ngLdUhcVexjloMvtx5766hG3ivmdCEeOPqjpA1AnwO8GEJKfnJnwzGdyYlrWw7cnUeimp8s490adGwWkR3gazMDOB8y3GT 2020-02-08T18:28:08.750 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-08T18:28:08.802 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-02-08T18:28:08.809 INFO:teuthology.task.internal:Saving configuration 2020-02-08T18:28:08.841 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-02-08T18:28:08.853 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-02-08T18:28:08.868 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-02-08 18:22:38.441437', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi136.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG9kfhByiBJTrTtV9+BjsBElnOTlcFYb5kTEVtWjaUIvO9l+LpOnLJPUm52pHI+xT51bmIKNHIEBnTDkvgyIfOM=', u'description': u'/home/teuthworker/archive/sage-2020-02-08_17:26:32-rados-wip-sage2-testing-2020-02-08-0831-distro-basic-smithi/4745670'} 2020-02-08T18:28:08.885 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-02-08 18:22:38.441453', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi121.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPNfORFjHnapcIhxkcQZ2nU1Xr0bTmqznkJUdRkTEiwVIg1Xkjp4e5/52gs0B3Ha28JCDMw11LYwn+Wufux28fQ=', u'description': u'/home/teuthworker/archive/sage-2020-02-08_17:26:32-rados-wip-sage2-testing-2020-02-08-0831-distro-basic-smithi/4745670'} 2020-02-08T18:28:08.900 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-02-08 18:22:38.441416', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi052.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwyOEh/5TuIJmMbGzdiYHJcY+CDa7pxsvY9sMgJzpnZ6Fusc69meq5xYH2WOGar3CEN0izN8au9dVe/U8MHulc=', u'description': u'/home/teuthworker/archive/sage-2020-02-08_17:26:32-rados-wip-sage2-testing-2020-02-08-0831-distro-basic-smithi/4745670'} 2020-02-08T18:28:08.901 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-02-08T18:28:08.914 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2020-02-08T18:28:08.914 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2020-02-08T18:28:08.915 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2020-02-08T18:28:08.915 INFO:teuthology.run_tasks:Running task console_log... 2020-02-08T18:28:09.038 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-02-08T18:28:09.038 INFO:teuthology.run_tasks:Running task internal.connect... 2020-02-08T18:28:09.049 INFO:teuthology.task.internal:Opening connections... 2020-02-08T18:28:09.049 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2020-02-08T18:28:09.051 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi121.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:28:09.179 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2020-02-08T18:28:09.181 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi136.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:28:09.296 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2020-02-08T18:28:09.297 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi052.front.sepia.ceph.com', 'timeout': 60} 2020-02-08T18:28:09.405 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-02-08T18:28:09.417 INFO:teuthology.orchestra.run.smithi121:> uname -m 2020-02-08T18:28:09.537 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2020-02-08T18:28:09.538 INFO:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2020-02-08T18:28:09.604 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Linux" 2020-02-08T18:28:09.604 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="7 (Core)" 2020-02-08T18:28:09.605 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2020-02-08T18:28:09.605 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2020-02-08T18:28:09.605 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="7" 2020-02-08T18:28:09.606 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-02-08T18:28:09.606 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2020-02-08T18:28:09.606 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-02-08T18:28:09.606 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.centos.org/" 2020-02-08T18:28:09.606 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-02-08T18:28:09.607 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-02-08T18:28:09.607 INFO:teuthology.orchestra.run.smithi121.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-02-08T18:28:09.607 INFO:teuthology.orchestra.run.smithi121.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-02-08T18:28:09.607 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-02-08T18:28:09.608 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-02-08T18:28:09.608 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-02-08T18:28:09.609 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2020-02-08T18:28:09.625 INFO:teuthology.orchestra.run.smithi136:> uname -m 2020-02-08T18:28:09.736 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2020-02-08T18:28:09.736 INFO:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2020-02-08T18:28:09.801 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Linux" 2020-02-08T18:28:09.802 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="7 (Core)" 2020-02-08T18:28:09.802 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2020-02-08T18:28:09.802 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2020-02-08T18:28:09.802 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="7" 2020-02-08T18:28:09.802 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-02-08T18:28:09.802 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2020-02-08T18:28:09.803 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-02-08T18:28:09.803 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.centos.org/" 2020-02-08T18:28:09.804 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-02-08T18:28:09.804 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:09.804 INFO:teuthology.orchestra.run.smithi136.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-02-08T18:28:09.804 INFO:teuthology.orchestra.run.smithi136.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-02-08T18:28:09.805 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-02-08T18:28:09.805 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-02-08T18:28:09.805 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:09.807 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2020-02-08T18:28:09.822 INFO:teuthology.orchestra.run.smithi052:> uname -m 2020-02-08T18:28:09.948 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2020-02-08T18:28:09.949 INFO:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2020-02-08T18:28:10.024 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Linux" 2020-02-08T18:28:10.024 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="7 (Core)" 2020-02-08T18:28:10.025 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2020-02-08T18:28:10.025 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2020-02-08T18:28:10.025 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="7" 2020-02-08T18:28:10.025 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2020-02-08T18:28:10.026 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2020-02-08T18:28:10.026 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2020-02-08T18:28:10.026 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.centos.org/" 2020-02-08T18:28:10.026 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-02-08T18:28:10.026 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:28:10.027 INFO:teuthology.orchestra.run.smithi052.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2020-02-08T18:28:10.027 INFO:teuthology.orchestra.run.smithi052.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2020-02-08T18:28:10.027 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-02-08T18:28:10.027 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2020-02-08T18:28:10.027 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:28:10.028 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2020-02-08T18:28:10.044 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-02-08T18:28:10.069 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-02-08T18:28:10.080 INFO:teuthology.task.internal:Checking for old test directory... 2020-02-08T18:28:10.080 INFO:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2020-02-08T18:28:10.084 INFO:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2020-02-08T18:28:10.089 INFO:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2020-02-08T18:28:10.180 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-02-08T18:28:10.191 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-02-08T18:28:10.192 INFO:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2020-02-08T18:28:10.196 INFO:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2020-02-08T18:28:10.199 INFO:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2020-02-08T18:28:10.321 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-02-08T18:28:10.391 INFO:teuthology.run_tasks:Running task kernel... 2020-02-08T18:28:10.410 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.8': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.1': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'osd.11': {'sha1': 'distro', 'kdb': True}, 'osd.10': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'osd.9': {'sha1': 'distro', 'kdb': True}, 'client.2': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}} 2020-02-08T18:28:10.410 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-02-08T18:28:10.411 INFO:teuthology.orchestra.run.smithi136:> uname -r 2020-02-08T18:28:10.498 INFO:teuthology.orchestra.run.smithi136.stdout:3.10.0-1062.4.1.el7.x86_64 2020-02-08T18:28:10.499 INFO:teuthology.task.kernel:Running kernel on smithi136: 3.10.0-1062.4.1.el7.x86_64 2020-02-08T18:28:10.499 INFO:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2020-02-08T18:28:11.392 INFO:teuthology.orchestra.run.smithi136.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-02-08T18:28:11.541 INFO:teuthology.orchestra.run.smithi136.stdout:Determining fastest mirrors 2020-02-08T18:28:13.936 INFO:teuthology.orchestra.run.smithi136.stdout: * base: mirror.ash.fastserv.com 2020-02-08T18:28:13.937 INFO:teuthology.orchestra.run.smithi136.stdout: * epel: ftp.linux.ncsu.edu 2020-02-08T18:28:13.938 INFO:teuthology.orchestra.run.smithi136.stdout: * extras: packages.oit.ncsu.edu 2020-02-08T18:28:13.938 INFO:teuthology.orchestra.run.smithi136.stdout: * updates: packages.oit.ncsu.edu 2020-02-08T18:28:16.768 INFO:teuthology.orchestra.run.smithi136.stdout:Resolving Dependencies 2020-02-08T18:28:16.768 INFO:teuthology.orchestra.run.smithi136.stdout:--> Running transaction check 2020-02-08T18:28:16.768 INFO:teuthology.orchestra.run.smithi136.stdout:---> Package kernel.x86_64 0:3.10.0-1062.12.1.el7 will be installed 2020-02-08T18:28:18.926 INFO:teuthology.orchestra.run.smithi136.stdout:--> Finished Dependency Resolution 2020-02-08T18:28:19.968 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:19.968 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies Resolved 2020-02-08T18:28:19.969 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:19.969 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2020-02-08T18:28:19.969 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2020-02-08T18:28:19.969 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2020-02-08T18:28:19.970 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2020-02-08T18:28:19.970 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 3.10.0-1062.12.1.el7 updates 50 M 2020-02-08T18:28:19.970 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:19.970 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2020-02-08T18:28:19.970 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2020-02-08T18:28:19.970 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2020-02-08T18:28:19.971 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:19.971 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 50 M 2020-02-08T18:28:19.971 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 64 M 2020-02-08T18:28:19.971 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading packages: 2020-02-08T18:28:19.993 INFO:teuthology.orchestra.run.smithi136.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-02-08T18:28:20.865 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2020-02-08T18:28:21.060 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2020-02-08T18:28:22.003 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded 2020-02-08T18:28:22.003 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2020-02-08T18:28:57.242 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-3.10.0-1062.12.1.el7.x86_64 1/1 2020-02-08T18:28:58.245 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-3.10.0-1062.12.1.el7.x86_64 1/1 2020-02-08T18:28:58.245 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:58.246 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2020-02-08T18:28:58.246 INFO:teuthology.orchestra.run.smithi136.stdout: kernel.x86_64 0:3.10.0-1062.12.1.el7 2020-02-08T18:28:58.246 INFO:teuthology.orchestra.run.smithi136.stdout: 2020-02-08T18:28:58.246 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2020-02-08T18:28:58.295 INFO:teuthology.orchestra.run.smithi136:> rpm -q kernel --last | head -n 1 2020-02-08T18:28:58.700 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-3.10.0-1062.12.1.el7.x86_64 Sat 08 Feb 2020 06:28:24 PM UTC 2020-02-08T18:28:58.703 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.12.1.el7.x86_64 2020-02-08T18:28:58.703 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.4.1.el7.x86_64 Expected: 3.10.0-1062.12.1.el7.x86_64 2020-02-08T18:28:58.703 INFO:teuthology.orchestra.run.smithi052:> uname -r 2020-02-08T18:28:58.814 INFO:teuthology.orchestra.run.smithi052.stdout:3.10.0-1062.4.1.el7.x86_64 2020-02-08T18:28:58.815 INFO:teuthology.task.kernel:Running kernel on smithi052: 3.10.0-1062.4.1.el7.x86_64 2020-02-08T18:28:58.815 INFO:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2020-02-08T18:28:59.656 INFO:teuthology.orchestra.run.smithi052.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-02-08T18:28:59.776 INFO:teuthology.orchestra.run.smithi052.stdout:Determining fastest mirrors 2020-02-08T18:29:00.404 INFO:teuthology.orchestra.run.smithi052.stdout: * base: mirror.ash.fastserv.com 2020-02-08T18:29:00.415 INFO:teuthology.orchestra.run.smithi052.stdout: * epel: ftp.linux.ncsu.edu 2020-02-08T18:29:00.416 INFO:teuthology.orchestra.run.smithi052.stdout: * extras: packages.oit.ncsu.edu 2020-02-08T18:29:00.417 INFO:teuthology.orchestra.run.smithi052.stdout: * updates: packages.oit.ncsu.edu 2020-02-08T18:29:03.603 INFO:teuthology.orchestra.run.smithi052.stdout:Resolving Dependencies 2020-02-08T18:29:03.603 INFO:teuthology.orchestra.run.smithi052.stdout:--> Running transaction check 2020-02-08T18:29:03.604 INFO:teuthology.orchestra.run.smithi052.stdout:---> Package kernel.x86_64 0:3.10.0-1062.12.1.el7 will be installed 2020-02-08T18:29:04.858 INFO:teuthology.orchestra.run.smithi052.stdout:--> Finished Dependency Resolution 2020-02-08T18:29:05.627 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:29:05.627 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies Resolved 2020-02-08T18:29:05.637 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:29:05.637 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2020-02-08T18:29:05.637 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2020-02-08T18:29:05.637 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2020-02-08T18:29:05.638 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2020-02-08T18:29:05.638 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 3.10.0-1062.12.1.el7 updates 50 M 2020-02-08T18:29:05.638 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:29:05.638 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2020-02-08T18:29:05.638 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2020-02-08T18:29:05.638 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2020-02-08T18:29:05.638 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:29:05.639 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 50 M 2020-02-08T18:29:05.639 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 64 M 2020-02-08T18:29:05.639 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading packages: 2020-02-08T18:29:05.656 INFO:teuthology.orchestra.run.smithi052.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-02-08T18:29:06.667 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2020-02-08T18:29:06.866 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2020-02-08T18:29:07.691 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded 2020-02-08T18:29:07.692 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2020-02-08T18:29:44.125 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-3.10.0-1062.12.1.el7.x86_64 1/1 2020-02-08T18:29:44.919 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-3.10.0-1062.12.1.el7.x86_64 1/1 2020-02-08T18:29:44.919 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:29:44.919 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2020-02-08T18:29:44.919 INFO:teuthology.orchestra.run.smithi052.stdout: kernel.x86_64 0:3.10.0-1062.12.1.el7 2020-02-08T18:29:44.920 INFO:teuthology.orchestra.run.smithi052.stdout: 2020-02-08T18:29:44.920 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2020-02-08T18:29:44.973 INFO:teuthology.orchestra.run.smithi052:> rpm -q kernel --last | head -n 1 2020-02-08T18:29:45.375 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-3.10.0-1062.12.1.el7.x86_64 Sat 08 Feb 2020 06:29:10 PM UTC 2020-02-08T18:29:45.377 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.12.1.el7.x86_64 2020-02-08T18:29:45.377 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.4.1.el7.x86_64 Expected: 3.10.0-1062.12.1.el7.x86_64 2020-02-08T18:29:45.377 INFO:teuthology.orchestra.run.smithi121:> uname -r 2020-02-08T18:29:45.498 INFO:teuthology.orchestra.run.smithi121.stdout:3.10.0-1062.4.1.el7.x86_64 2020-02-08T18:29:45.498 INFO:teuthology.task.kernel:Running kernel on smithi121: 3.10.0-1062.4.1.el7.x86_64 2020-02-08T18:29:45.499 INFO:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2020-02-08T18:29:46.429 INFO:teuthology.orchestra.run.smithi121.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2020-02-08T18:29:46.569 INFO:teuthology.orchestra.run.smithi121.stdout:Determining fastest mirrors 2020-02-08T18:29:46.964 INFO:teuthology.orchestra.run.smithi121.stdout: * base: mirror.ash.fastserv.com 2020-02-08T18:29:46.975 INFO:teuthology.orchestra.run.smithi121.stdout: * epel: ftp.linux.ncsu.edu 2020-02-08T18:29:46.975 INFO:teuthology.orchestra.run.smithi121.stdout: * extras: packages.oit.ncsu.edu 2020-02-08T18:29:46.976 INFO:teuthology.orchestra.run.smithi121.stdout: * updates: packages.oit.ncsu.edu 2020-02-08T18:29:50.182 INFO:teuthology.orchestra.run.smithi121.stdout:Resolving Dependencies 2020-02-08T18:29:50.183 INFO:teuthology.orchestra.run.smithi121.stdout:--> Running transaction check 2020-02-08T18:29:50.183 INFO:teuthology.orchestra.run.smithi121.stdout:---> Package kernel.x86_64 0:3.10.0-1062.12.1.el7 will be installed 2020-02-08T18:29:51.603 INFO:teuthology.orchestra.run.smithi121.stdout:--> Finished Dependency Resolution 2020-02-08T18:29:52.509 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-02-08T18:29:52.509 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies Resolved 2020-02-08T18:29:52.514 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-02-08T18:29:52.514 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2020-02-08T18:29:52.514 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2020-02-08T18:29:52.514 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2020-02-08T18:29:52.514 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2020-02-08T18:29:52.514 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 3.10.0-1062.12.1.el7 updates 50 M 2020-02-08T18:29:52.514 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-02-08T18:29:52.515 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2020-02-08T18:29:52.515 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2020-02-08T18:29:52.515 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2020-02-08T18:29:52.515 INFO:teuthology.orchestra.run.smithi121.stdout: 2020-02-08T18:29:52.516 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 50 M 2020-02-08T18:29:52.516 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 64 M 2020-02-08T18:29:52.516 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading packages: 2020-02-08T18:29:52.524 INFO:teuthology.orchestra.run.smithi121.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2020-02-08T18:29:53.448 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2020-02-08T18:29:53.657 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2020-02-08T18:29:54.522 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded 2020-02-08T18:29:54.522 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2020-02-08T18:29:57.409 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-02-08T18:29:57.410 DEBUG:teuthology.task.console_log:Killing console logger for smithi136 2020-02-08T18:29:57.410 DEBUG:teuthology.task.console_log:Killing console logger for smithi052 2020-02-08T18:29:57.410 DEBUG:teuthology.task.console_log:Killing console logger for smithi121 2020-02-08T18:29:57.411 DEBUG:teuthology.task.console_log:Killing console logger for smithi121 2020-02-08T18:29:57.411 DEBUG:teuthology.task.console_log:Killing console logger for smithi052 2020-02-08T18:29:57.420 DEBUG:teuthology.task.console_log:Killing console logger for smithi136 2020-02-08T18:29:57.421 DEBUG:teuthology.exit:Finished running handlers