搭建spark集群
搭建spark集群 spark1.6和hadoop2.6
1.准备hadoop环境；
2.准备下载包；
3.解压安装包：
tar -xf spark-1.6.0-bin-hadoop2.6.tgz
4.修改配置文件
目录：/apps/app/spark1.6-hadoop2.6/conf
a.修改spark-env.sh:
cp spark-env.sh.template spark-env.sh
vim spark-env.sh
添加：
export JAVA_HOME=/apps/app/dk/jdk1.7.0_79
export SCALA_HOME=/apps/app/dk/scala-2.10.4
export SPARK_DRIVER_MEMORY=1g
export SPARK_WORKER_MEMORY=512m
export SPARK_WORKER_CORES=2
export SPARK_EXECUTOR_MEMORY=512m
export SPARK_EXECUTOR_CORES=2
export HADOOP_HOME=/apps/app/hadoop-2.6.0
export HADOOP_CONF_DIR=/apps/app/hadoop-2.6.0/etc/hadoop
b.修改spark-defaults.conf
cp spark-defaults.conf.template spark-defaults.conf
vim spark-defaults.conf
spark.eventLog.enabled true
spark.eventLog.dir hdfs://kylin01:9000/historyserverforspark
spark.history.fs.logDirectory hdfs://kylin01:9000/historyserverforspark
spark.yarn.historyServer.address kylin01:18880
spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
c.修改：hive-site.xml
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://kylin06:9083</value>
</property>
<property>
<name>hive.server2.thrift.min.worker.threads</name>
<value>5</value>
<description>Minimum number of Thrift worker threads</description>
</property>
<property>
<name>hive.server2.thrift.max.worker.threads</name>
<value>50</value>
<description>Maximum number of Thrift worker threads</description>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
<description>Port number of HiveServer2 Thrift interface. Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT</description>
</property>
<property>
<name>hive.server2.thrift.bind.host</name>
<value>kylin06</value>
<description>Bind host on which to run the HiveServer2 Thrift interface.Can be overridden by setting$HIVE_SERVER2_THRIFT_BIND_HOST</description>
</property>
</configuration>
d.修改:slaves
cp slaves.template slaves
vim slaves
视频地址：http://v.qq.com/x/search/?q=%E5%A4%A7%E6%95%B0%E6%8D%AE%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA&stag=0&smartbox_ab=
