Spark—local模式环境搭建

Spark——local模式环境搭建

一、Spark运行模式介绍

1、本地模式（loca模式）：spark单机运行，一般用户测试和开发使用

2、Standalone模式：构建一个主从结构(Master+Slave)的spark集群,spark运行在集群中。

3、Spark on yarn 模式：Spark客户端直接连接Yarn,不用构建Spark集群

4、Spark on Mesos 模式：Spark客户端直接连接Mesos.不需要额外构建Spark集群

二、local模式

1、将编译好的spark包解压到指定目录，我这里是使用spark源码编译的，编译过程看上篇博客

tar -zxvf ./spark-2.1.0-bin-2.6.0-cdh5.7.0.tgz -C /home/hadoop/app/

2、配置环境变量

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131
export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export HIVE_HOME=/home/hadoop/app/hive-1.1.0-cdh5.7.0
export MAVEN_HOME=/home/hadoop/app/apache-maven-3.5.4
export SCALA_HOME=/home/hadoop/app/scala-2.11.8
export SPARK_HOME=/home/hadoop/app/spark-2.1.0-bin-2.6.0-cdh5.7.0
export PATH=$SPARK_HOME/bin:$SCALA_HOME/bin:$MAVEN_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$JAVA_HOME/bin:
$PATH

3、启动local模式

spark-shell --master local[2]

启动过程如下：

[hadoop@hadoop01 ~]$ spark-shell --master local[2]
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/10/09 19:49:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... 
using builtin-java classes where applicable
18/10/09 19:50:13 WARN ObjectStore: Version information not found in metastore. 
hive.metastore.schema.verification is not enabled so recording 
the schema version 1.2.0
18/10/09 19:50:13 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
18/10/09 19:50:15 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.44.183:4040
Spark context available as 'sc' (master = local[2], app id = local-1539085800463).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
      /_/
 
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.
 
scala>