Tensorflowonspark安装
1.实验环境
Centos7+Python3.6+Java8+Hadoop2.6+Spark2.3+Tensorflow1.10.0
2.Tensorflow安装
最简单的方式:pip install tensorflow==1.9
测试tf,无异常说明安装成功
import tensorflow as tf
3.下载TensorflowOnSpark源码
git clone https://github.com/yahoo/TensorFlowOnSpark.git
cd TensorFlowOnSpark
export TFoS_HOME=$(pwd)
下载成功后,你会得到类似上面的文件夹,tfspark.zip是我们生成的python库文件,之后提交Spark的时候用到,其就是把tensorflowonspark所有文件进行了打包,在TensorFlowOnSpark目录运行如下的命令进行打包(Keng4)
zip -r tfspark.zip tensorflowonspark/*
[root@master TensorFlowOnSpark]# ls
examples LICENSE README.md scripts setup.cfg setup.py tensorflow tensorflowonspark tfspark.zip
4. 数据准备
我们以Fashion MNIST数据集为例,介绍生成TFRecrd的方法。
下面我们把数据集下载并保存到data/fashion目录下:
$ mkdir -p data/fashin
$ cd data/fashion
$ wget http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
$ wget http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz
$ wget http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz
$ wget http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz
$ cd ../..
5.Spark集群测试
1>转换MNIST数据文件
${SPARK_HOME}/bin/spark-submit \
--master=local[*] \
${TFoS_HOME}/examples/mnist/mnist_data_setup.py \
--output examples/mnist/csv \
--format csv