在centos7上安装部署hadoop2.7.3和spark2.0.0

一、安装装备

下载安装包：

vmware workstations pro 12

三台centos7.1 mini 虚拟机

网络配置NAT网络如下：

二、创建hadoop用户和hadoop用户组

1. groupadd hadoop

2. useradd hadoop

3. 给hadoop用户设置密码

    在root用户下：passwd hadoop设置新密码

三、关闭防火墙和selinux

1. yum install -y firewalld

2. systemctl stop firewalld

3. systemctl disable firewalld

4. vi /etc/selinux/config

5.全部设置好重启虚拟机

四、三台虚拟机之间ssh互通

1. 在hadoop用户下

1. ssh-keygen -t rsa

2. ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub hadoop@虚拟机ip

3. ssh namenode/dnode1/dnode2

五、安装Java

1. 官网下载jdk1.8.rpm包

2.  rpm -ivh jdk1.8.rpm

六、安装hadoop

1. 官网下载hadoop2.7.3.tar.gz

2. tar xzvf hadoop2.7.3.tar.gz

3. mv hadoop2.7.3 /usr/local/

4. chown -R hadoop:hadoop /usr/local/hadoop-2.7.3

七、环境变量配置

1.vim /home/hadoop/.bash_profile

source /home/hadoop/.bash_profile

export JAVA_HOME=/usr/java/jdk1.8.0_101
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=/usr/local/hadoop-2.7.3
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SCALA_HOME=/usr/local/scala-2.11.8
export SPARK_HOME=/usr/local/spark-2.0.0
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin:$SPARK_HOME/bin
export PATH

八、建立hadoop相关目录

mkdir -p /home/hadoop/hd_space/tmp
mkdir -p /home/hadoop/hd_space/hdfs/name
mkdir -p /home/hadoop/hd_space/hdfs/data
mkdir -p /home/hadoop/hd_space/mapred/local
mkdir -p /home/hadoop/hd_space/mapred/system
chown -R hadoop:hadoop /home/hadoop

注意：至此，可以克隆虚拟机，不必每台都安装centos，克隆之后再建立互通

九、配置hadoop

1. 配置core-site.xml

    vim /usr/local/hadoop-2.7.3/etc/hadoop/core-site.xml

<configuration>
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://namenode:9000</value>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/hd_space/tmp</value>
</property>
<property>
    <name>io.file.buffer.size</name>
    <value>131702</value>
</property>
</configuration>

2. 配置hdfs-site.xml

<property>
        <name>dfs.namenode.name.dir</name>
        <value>/home/hadoop/hd_space/hdfs/name</value>?
</property>
<property>
        <name>dfs.datanode.data.dir</name>
        <value>/home/hadoop/hd_space/hdfs/data</value>
</property>
<property>
        <name>dfs.replication</name>
        <value>2</value>
</property>
<property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>dnode1:50090</value>
</property>
<property>
        <name>dfs.namenode.secondary.https-address</name>
        <value>dnode1:50091</value>
</property>

3. 配置mapred-site.xml

<configuration>
<property>
        <name>mapreduce.cluster.local.dir</name>
        <value>/home/hadoop/hd_space/mapred/local</value>
</property>
<property>
        <name>mapreduce.cluster.system.dir</name>
        <value>/home/hadoop/hd_space/mapred/system</value>
</property>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>namenode:10020</value>
</property>?
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>namenode:19888</value>
</property>?
</configuration>

4.配置yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
<property>
        <description>The?hostname?of?the?RM.</description>
        <name>yarn.resourcemanager.hostname</name>
        <value>namenode</value>
</property>
<property>
        <description>the?valid?service?name?should?only?contain?a-zA-Z0-9_?and?can?not?start?with?numbers</description>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>?
</configuration>

5. 配置slaves，两个数据节点dnode1，dnode2

6. 拷贝配置文件目录到另外两台数据节点

for target in dnode1 dnode2 do  
    scp -r /usr/local/hadoop-2.7.3/etc/hadoop $target:/usr/local/hadoop-2.7.3/etc
done 

十、启动hadoop

1. 用户登陆hadoop

2. 格式化hdfs

hdfs namenode -format

3. 启动dfs

start-dfs.sh

4. 启动yarn

start-yarn.sh

十一、错误解决

1. 出现以下环境变量JAVA_HOME找不到。

解决办法：需要重新配置/usr/local/hadoop-2.7.3/libexec/hadoop-config.sh

2. oracle数据出现以下错误解决

TNS-12555: TNS:permission denied，监听器启动不了 lsnrctl start（启动）/status（查看状态）/stop（停止）

解决办法：chown -R hadoop:hadoop /var/tmp/.oracle

chmod 777 /var/tmp/.oracle

问题：

十二、客户端验证

1. jps检查进程

至此，hadoop已经安装完毕

十三、测试hadoop，hadoop用户登陆

    运行hadoop自带的wordcount实例

1. 建立输入文件

mkdir -p -m 755 /home/hadoop/test

cd /home/hadoop/test

echo "My first hadoop example. Hello Hadoop in input. " > testfile.txt

2. 建立目录

hadoop fs -mkdir /test

#hadoop fs -rmr /test 删除目录

3. 上传文件

hadoop fs -put testfile.txt /test

4. 执行wordcount程序

hadoop jar /usr/local/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /test/testfile.txt /test/output

5. 查看结果

hadoop fs -cat /test/output/part-r-00000

十四、安装Spark2.0.0

1. 解压scala-2.11.8.tar.gz到/usr/local

tar xzvf scala-2.11.8 

mv scala-2.11.8 /usr/local

2. 解压spark-2.0..0.tgz 到/usr/local

tar xzvf spark-2.0.0.tar.gz 

mv spark-2.0.0 /usr/local

3.配置spark

cd /usr/local/spark-2.0.0/conf

vim spark-env.sh

export JAVA_HOME=/usr/java/jdk1.8.0_101
export SCALA_HOME=/usr/local/scala-2.11.8
export SPARK_HOME=/usr/local/spark-2.0.0
export SPARK_MASTER_IP=namenode
export SPARK_WORKER_MEMORY=1024m
export master=spark://namenode:7070

vim slaves

同步资源及配置文件到其它两个节点（hadoop用户）

for target in dnode1 dnode2 do  
    scp -r /usr/local/scala-2.11.8 $target:/usr/local
    scp -r /usr/local/spark-2.0.0 $target:/usr/local 
done

4.启动spark集群

cd $SPARK_HOME 

# Start Master 

./sbin/start-master.sh  

# Start Workers 

./sbin/start-slaves.sh

5.客户端验证