spark（一） build

（1）编译前的准备工作，安装jdk，解压maven，解压spark，解压scala并配置相关的环境变量
export JAVA_HOME=/opt/module/jdk1.6.0_45
export PATH=$PATH:$JAVA_HOME/bin

export MAVEN_HOME=/opt/module/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin

export SCALA_HOME=/opt/module/scala-2.10.4
export PATH=$PATH:$SCALA_HOME/bin
（2）如果使用mvn方式，还需增加 export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"，不然会出现错误错误
一切工作准备就绪，则开始编译 mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
这些数值设置的是不通的参数的值，可以查看官方文档，根据提示设置其他的值。
（3）如果使用的是make-distribution.sh，则需要做以下修改
配置域名服务器
#vim /resolv.conf
内容：
nameserver 8.8.8.8
nameserver 8.8.4.4
最后：
./make-distribution.sh --tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive-0.13.1 -Phive-thriftserver
./make-distribution.sh --tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive-0.13.1 -Phive-thriftserver
为了减少时间，我们对make-distribution.sh文件做如下修改
VERSION=1.3.0 #设置spark版本
SPARK_HADOOP_VERSION=2.6.0-cdh5.4.0
SPARK_HIVE=1
等待1个半小时 。。。。。。。。。。。。。。。。
出现了下边的内容，则成功了
Saving to outputFile=/opt/module/spark-1.3.0/external/kafka-assembly/scalastyle-output.xml
Processed 0 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 0 ms
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM .......................... SUCCESS [20.658s]
[INFO] Spark Project Networking .......................... SUCCESS [59.540s]
[INFO] Spark Project Shuffle Streaming Service ........... SUCCESS [19.474s]
[INFO] Spark Project Core ................................ SUCCESS [17:10.312s]
[INFO] Spark Project Bagel ............................... SUCCESS [3:25.347s]
[INFO] Spark Project GraphX .............................. SUCCESS [6:48.707s]
[INFO] Spark Project Streaming ........................... SUCCESS [4:51.339s]
[INFO] Spark Project Catalyst ............................ SUCCESS [6:09.566s]
[INFO] Spark Project SQL ................................. SUCCESS [5:40.546s]
[INFO] Spark Project ML Library .......................... SUCCESS [6:50.022s]
[INFO] Spark Project Tools ............................... SUCCESS [1:02.293s]
[INFO] Spark Project Hive ................................ SUCCESS [7:17.840s]
[INFO] Spark Project REPL ................................ SUCCESS [2:37.665s]
[INFO] Spark Project YARN ................................ SUCCESS [1:23.802s]
[INFO] Spark Project Assembly ............................ SUCCESS [3:10.951s]
[INFO] Spark Project External Twitter .................... SUCCESS [2:56.559s]
[INFO] Spark Project External Flume Sink ................. SUCCESS [1:07.428s]
[INFO] Spark Project External Flume ...................... SUCCESS [34.766s]
[INFO] Spark Project External MQTT ....................... SUCCESS [1:47.120s]
[INFO] Spark Project External ZeroMQ ..................... SUCCESS [38.234s]
[INFO] Spark Project External Kafka ...................... SUCCESS [1:05.513s]
[INFO] Spark Project Examples ............................ SUCCESS [7:23.760s]
[INFO] Spark Project YARN Shuffle Service ................ SUCCESS [40.643s]
[INFO] Spark Project External Kafka Assembly ............. SUCCESS [48.520s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:25:14.796s
[INFO] Finished at: Mon Nov 16 23:55:37 PST 2015
[INFO] Final Memory: 65M/200M
[INFO] ------------------------------------------------------------------------