Hadoop NameNode HA(验证成功)

name node1 JournalNode1: 192.168.81.132 -> hadoop1

name node2 JournalNode2: 192.168.81.130 -> hadoop2

data1 node JournalNode3: 192.168.81.129 -> hadoop3

data2 node: 192.168.81.131 -> hadoop4

一、创建账号

1.所有节点创建用户 useradd hadoop passwd hadoop

2.所有节点创建目录 mkdir -p /home/hadoop/source mkdir -p /home/hadoop/tools

3.Slave节点创建目录 mkdir -p /hadoop/hdfs mkdir -p /hadoop/tmp mkdir -p /hadoop/log chmod -R 777 /hadoop

二、修改主机名

所有节点修改

1.vim /etc/sysconfig/network ，修改 HOSTNAME=hadoopx

2.vim /etc/hosts

192.168.81.132 hadoop1

192.168.81.130 hadoop2

192.168.81.129 hadoop3

192.168.81.131 hadoop4

3.执行 hostname hadoopx

4.重新登录，即可

三、免密码登录

注意：非root用户免密码登录，需要执行 chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys 如果不修改权限，非root用户无法免密码登录

四、安装JDK

五、配置环境变量

1. /etc/profile

export HADOOP_HOME=/home/hadoop/hadoop

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

export HADOOP_MAPARED_HOME=${HADOOP_HOME}

export HADOOP_COMMON_HOME=${HADOOP_HOME}

export HADOOP_HDFS_HOME=${HADOOP_HOME}

export YARN_HOME=${HADOOP_HOME}

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

2. hadoop-env.sh 在末尾添加 export JAVA_HOME=/usr/java/jdk1.6.0_27

六、Hadoop 2.3安装

1.core-site.xml

<name>hadoop.tmp.dir</name>

<value>/home/hadoop/tmp</value>

<description>A base for other temporary directories.</description>

</property>

<name>fs.default.name</name>

<value>hdfs://hadoop1:9000</value>

</property>

</configuration>

2.添加slaves节点 hadoop3 hadoop4

3.配置hdfs-site.xml

<property> <name>dfs.replication</name> <value>2</value> </property>

<property> <name>dfs.nameservices</name> <value>cluster1</value> </property>

<property> <name>dfs.ha.namenodes.cluster1</name> <value>hadoop1,hadoop2</value> </property>

<property> <name>dfs.namenode.rpc-address.cluster1.hadoop1</name> <value>hadoop1:9000</value> </property>

<property> <name>dfs.namenode.rpc-address.cluster1.hadoop2</name> <value>hadoop2:9000</value> </property>

<property> <name>dfs.namenode.http-address.cluster1.hadoop1</name> <value>hadoop1:50070</value> </property>

<property> <name>dfs.namenode.http-address.cluster1.hadoop2</name> <value>hadoop2:50070</value> </property>

<property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/cluster1</value> </property>

<property> <name>dfs.client.failover.proxy.provider.cluster1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property>

<property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property>

<property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/root/.ssh/id_rsa</value> </property>

<property> <name>dfs.journalnode.edits.dir</name> <value>/hadoop/tmp/journal</value> </property>

<property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property>

</configuration>
4.配置yarn-site.xml

<property> <name>yarn.resourcemanager.address</name> <value>192.168.81.132:18040</value> </property>

<property> <name>yarn.resourcemanager.scheduler.address</name> <value>192.168.81.132:18030</value> </property>

<property> <name>yarn.resourcemanager.webapp.address</name> <value>192.168.81.132:18088</value> </property>

<property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>192.168.81.132:18025</value> </property>

<property> <name>yarn.resourcemanager.admin.address</name> <value>192.168.81.132:18141</value> </property>

<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>

<property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> </configuration>
mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>192.168.81.132:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>192.168.81.132:19888</value> </property> </configuration>
5.配置httpfs-site.xml 略

七、配置DataNode

1.复制hadoop-2.3至各Data节点

2.复制环境变量/etc/profile，生效

八、启动并验证

1.启动JournalNode集群

在hadoop1,hadoop2,hadoop3

执行 hadoop-daemon.sh start journalnode

jps 是否有journalnode进程

2.格式化并启动主NameNode hadoop1

hdfs namenode -format

hadoop-daemon.sh start namenode

jps 是否有namenode进程