Uubntu14.04下 Hadoop集群环境搭建
1机器信息
master192.168.3.10 hadoop/ hadoop
slave1192.168.3.11 hadoop/ hadoop
slave2192.168.3.12 hadoop/ hadoop
（1）创建用户
#sudo adduser hadoop
（2）增加root权限
# vim /etc/sudoers
hadoop ALL=(ALL) ALL
（3）修改以上三台主机的/etc/hosts配置信息（内容一样）
192.168.3.10 hd-master
192.168.3.11 hd-slave1
192.168.3.12 hd-slave2
（4）分别修改/etc/hostname配置文件
2设置master到slave节点的SSH无密码登陆
Ubuntu已经默认安装了SSH
查看是否运行：ps -ef | grep sshd
（1）首先生成hd-master公钥
hadoop@hd-master:~$ ssh localhost
hadoop@hd-master:~/.ssh$ ssh-keygen -t rsa
hadoop@hd-master:~/.ssh$ cat id_rsa.pub >> authorized_keys
（2）将hd-master公钥保存到hd-slave1节点
hadoop@hd-master:~/.ssh$ scp id_rsa.pub hadoop@hd-slave1:/home/hadoop/
（3）在 hd-slave1节点 上将ssh公匙保存到相应位置
hadoop@hd-slave1:~$ cat id_rsa.pub >> ~/.ssh/authorized_keys
（4）同样将hd-master公钥保存到hd-slave2节点，并在hd-slave2节点 上将ssh公匙保存到相应位置
hadoop@hd-master:~/.ssh$ scp id_rsa.pub hadoop@hd-slave2:/home/hadoop/
hadoop@hd-slave2:~$ cat id_rsa.pub >> ~/.ssh/authorized_keys
（5）最后就可以在hd-master节点上就可以无密码SSH登录到hd-slave1、hd-slave2
hadoop@hd-master:~/.ssh$ ssh hd-slave1
hadoop@hd-master:~/.ssh$ ssh hd-slave2
3安装JDK
hadoop@hd-master:~$ java -version
java version "1.8.0_91"
hadoop@hd-slave1:~$ java -version
java version "1.8.0_91"
hadoop@hd-slave2:~$ java -version
java version "1.7.0_40"
4关闭防火墙
查看防护墙状态是否为“不活动”
hadoop@hd-master:~$ sudo ufw status
Status: inactive
5安装hadoop-2.7.3.tar.gz
（1）解压
hadoop@hd-master:~$ tar -zxvf hadoop-2.7.3.tar.gz
（2）在hd-master节点创建以下目录，并属于同一用户及用户组
hadoop@hd-master:~$ mkdir -p ~/dfs/name && mkdir ~/dfs/data && mkdir ~/tmp
（3）在hd-master节点修改配置文件
A: hadoop-env.sh
export JAVA_HOME=${JAVA_HOME}
B: yarn-env.sh
export JAVA_HOME=${JAVA_HOME}
C: slaves保存所有slave节点
hd-slave1
hd-slave2
D: core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://hd-master:8010</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
E: hdfs-site.xml
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hd-master:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
F: mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hd-master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hd-master:19888</value>
</property>
G: yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hd-master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hd-master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hd-master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hd-master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hd-master:8088</value>
</property>
（4）将hd-master节点中的hadoop-2.7.3/复制到hd-slave1与hd-slave2节点。
hadoop@hd-master:~$ scp -r hadoop-2.7.3 hadoop@hd-slave1:/home/hadoop/
hadoop@hd-master:~$ scp -r hadoop-2.7.3 hadoop@hd-slave2:/home/hadoop/
（5）配置环境变量
hadoop@hd-master:~$ cat /etc/environment
PATH="/home/hadoop/hadoop-2.7.3/bin:/home/hadoop/hadoop-2.7.3/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
（6）格式化namenode节点
hadoop@hd-master:~$ hdfs namenode -format
（7）启动hdfs
hadoop@hd-master:~$ start-dfs.sh
Starting namenodes on [hd-master]
hd-master: namenode running as process 7177. Stop it first.
hd-slave1: datanode running as process 2958. Stop it first.
hd-slave2: datanode running as process 10759. Stop it first.
Starting secondary namenodes [hd-master]
hd-master: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-secondarynamenode-hd-master.out
（8）启动yarn
hadoop@hd-master:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop-2.7.3/logs/yarn-hadoop-resourcemanager-hd-master.out
hd-slave1: starting nodemanager, logging to /home/hadoop/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-hd-slave1.out
hd-slave2: starting nodemanager, logging to /home/hadoop/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-hd-slave2.out
hadoop@hd-master:~$
（9）查看进程
hadoop@hd-master:~$ jps
8324 ResourceManager
7177 NameNode
8667 Jps
8173 SecondaryNameNode
hadoop@hd-slave1:~$ jps
3335 NodeManager
3452 Jps
2958 DataNode
hadoop@hd-slave2:~$ jps
10759 DataNode
13192 Jps
12529 NodeManager
（10）查看集群信息
http://192.168.3.10:8088/cluster
1机器信息
|
master |
192.168.3.10 hadoop/ hadoop |
|
slave1 |
192.168.3.11 hadoop/ hadoop |
|
slave2 |
192.168.3.12 hadoop/ hadoop |
