总访问量: PV

DataScience && DataMining && BigData

spark-sklearn(spark扩展scikitlearn)

(1)官方规定安装条件:此包装具有以下要求:

-*最新版本的scikit学习。 版本0.17已经过测试,旧版本也可以使用。
- *Spark> = 2.0。 Spark可以从对应官网下载
[Spark官方网站](http://spark.apache.org/)

-*为了使用spark-sklearn,您需要使用pyspark解释器或其他Spark兼容的python解释器。

有关详细信息,请参阅[Spark指南](https://spark.apache.org/docs/latest/programming-guide.html#overview)。
- (https://nose.readthedocs.org)(仅测试依赖关系)

英文原文:This package has the following requirements:
- a recent version of scikit-learn. Version 0.17 has been tested, older versions may work too.
- Spark >= 2.0. Spark may be downloaded from the
[Spark official website](http://spark.apache.org/) In order to use spark-sklearn, you need to use the pyspark interpreter or another Spark-compliant python interpreter. See the [Spark guide](https://spark.apache.org/docs/latest/programming-guide.html#overview) for more details.
- [nose](https://nose.readthedocs.org) (testing dependency only)

(2)首先安装pyspark:

参考为的博客:http://www.cnblogs.com/jackchen-Net/p/6667205.html#_label5

(3)访问网址:https://pypi.python.org/pypi/spark-sklearn

目前Spark集成了Scikit-learn包,这样可以极大的简化了python数据科学家们的工作,这个包可以在Spark集群上自动分配模型参数优化计算任务

 (4)官方文档的例子测试

 1 ## Example
 2 
 3 Here is a simple example that runs a grid search with Spark. See the [Installation](#Installation) section on how to install spark-sklearn.
 4 
 5 ```python
 6 from sklearn import svm, grid_search, datasets
 7 from spark_sklearn import GridSearchCV
 8 iris = datasets.load_iris()
 9 parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
10 svr = svm.SVC()
11 clf = GridSearchCV(sc, svr, parameters)
12 clf.fit(iris.data, iris.target)
13 ```
14 
15 This classifier can be used as a drop-in replacement for any scikit-learn classifier, with the same API.

END~

posted @ 2017-08-07 09:23  CJZhaoSimons  阅读(4414)  评论(0编辑  收藏  举报