HDFS NameNode HA 部署文档
简介：
HDFS High Availability Using the Quorum Journal Manager
Hadoop 2.x 中，HDFS 组件有三个角色：NameNode、DataNode、Secondary NameNode
1、NameNode
保存着 HDFS 的元数据信息，包括命名空间、块信息等。
维护着两种文件：fsimage、edit logs
2、DataNode
保存着 HDFS 的数据信息。
3、Secondary NameNode
负责合并 edit logs 到 fsimage。
# NameNode 会将元数据信息 fsimage 加载到内存中，供 DataNode 使用，将每次的修改保存在 edit logs 中。
# 每次 NameNode 重启时，将 edit logs 合并到 fsimage 中。
# 然而生产环境中 NameNode 很少重启，这就导致 edit logs 文件无限变大，当重启 NameNode 时，合并时间变长、启动时间变长。
# Secondary NameNode 负责定期将 edit logs 合并到 fsimage 然后同步给 NameNode，避免上述问题。
Hadoop 2.x 默认情况下，整个集群只有一台服务器充当 NameNode 角色，如果 NameNode 宕机，则 HDFS 及所有依赖 HDFS 的组件都无法访问。
由此可见 NameNode 的重要性，所以需要给 NameNode 做 HA ！
角色分配如下： 192.168.1.25 master.hadoop (NameNode Active 、ZKFC) 192.168.1.26 backup.hadoop (NameNode Standby、ZKFC) 192.168.1.27 datanode01.hadoop (DataNode、ZooKeeper Server、JournalNode) 192.168.1.28 datanode02.hadoop (DataNode、ZooKeeper Server、JournalNode) 192.168.1.29 datanode03.hadoop (DataNode、ZooKeeper Server、JournalNode)
一、搭建 Hadoop 集群
文档链接：http://www.cnblogs.com/wangxiaoqiangs/p/6700717.html
# 停止集群 sh sbin/stop-all.sh
二、搭建 ZooKeeper 集群
文档链接：http://www.cnblogs.com/wangxiaoqiangs/p/6809065.html
shell > ansible datanode -m shell -a '/usr/local/zookeeper-3.4.10/bin/zkServer.sh status' datanode02.hadoop | SUCCESS | rc=0 >> Mode: followerZooKeeper JMX enabled by default Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg datanode03.hadoop | SUCCESS | rc=0 >> Mode: leaderZooKeeper JMX enabled by default Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg datanode01.hadoop | SUCCESS | rc=0 >> Mode: followerZooKeeper JMX enabled by default Using config: /usr/local/zookeeper-3.4.10/bin/../conf/zoo.cfg shell > sh bin/zkCli.sh -server 192.168.1.27:2181,192.168.1.28:2181,192.168.1.29:2181 [zk: 192.168.1.27:2181,192.168.1.28:2181,192.168.1.29:2181(CONNECTED) 0] ls / [zookeeper]
# 验证集群状态
三、配置 HDFS NameNode HA
1、修改 core-site.xml
shell > vim etc/hadoop/core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://myhdfs</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:///data/hadoop/tmp</value> </property> <property> <name>fs.trash.interval</name> <value>1440</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>192.168.1.27:2181,192.168.1.28:2181,192.168.1.29:2181</value> </property> <property> <name>ha.zookeeper.session-timeout.ms</name> <value>5000</value> </property> </configuration>
# fs.defaultFS 指定的不再是某台 NameNode 的地址，是一个逻辑名称 ( hdfs-site.xml 文件中定义 )
# ha.zookeeper.quorum 指定 ZooKeeper Server 地址
# ha.zookeeper.session-timeout NameNode 与 ZooKeeper Server 超时时间，超时会发生主备切换
2、修改 hdfs-site.xml
shell > vim etc/hadoop/hdfs-site.xml <configuration> <property> <name>dfs.blocksize</name> <value>134217728</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:///data/dfs/nn</value> </property> <property> <name>dfs.namenode.checkpoint.dir</name> <value>file:///data/dfs/sn</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///data/dfs/dn</value> </property> <property> <name>dfs.namenode.handler.count</name> <value>20</value> </property> <property> <name>dfs.nameservices</name> <value>myhdfs</value> </property> <property> <name>dfs.ha.namenodes.myhdfs</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.myhdfs.nn1</name> <value>master.hadoop:8020</value> </property> <property> <name>dfs.namenode.rpc-address.myhdfs.nn2</name> <value>backup.hadoop:8020</value> </property> <property> <name>dfs.namenode.http-address.myhdfs.nn1</name> <value>master.hadoop:50070</value> </property> <property> <name>dfs.namenode.http-address.myhdfs.nn2</name> <value>backup.hadoop:50070</value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://192.168.1.27:8485;192.168.1.28:8485;192.168.1.29:8485/journal</value> </property> <property> <name>dfs.client.failover.proxy.provider.myhdfs</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/data/dfs/journal</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/hadoop/.ssh/id_rsa</value> </property> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>5000</value> </property> </configuration>
四、初始化 backup.hadoop
# 根据集群部署文档来做 ( 修改主机名、配置hosts、创建hadoop用户、密钥认证、java环境、修改/data目录权限等 )
# 密钥认证要与原 NameNode、DataNode 都做
五、同步配置文件
1、修改 HOSTS
shell > vim /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.25 master.hadoop 192.168.1.26 backup.hadoop 192.168.1.27 datanode01.hadoop 192.168.1.28 datanode02.hadoop 192.168.1.29 datanode03.hadoop
2、密钥认证
shell > ssh-copy-id -i ~/.ssh/id_rsa.pub "-p 22 root@backup.hadoop" shell > su - hadoop hadoop shell > ssh-copy-id -i ~/.ssh/id_rsa.pub "-p 22 hadoop@backup.hadoop"
3、同步配置文件
shell > ansible all -m synchronize -a 'src=/usr/local/hadoop-2.8.0 dest=/usr/local/'
六、初始化、启动 NameNode HA
1、初始化 zkfc
shell > su - hadoop hadoop shell > hdfs zkfc -formatZK INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/myhdfs in ZK.
2、启动 journalnode
hadoop shell > ansible datanode -m shell -a '/usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start journalnode'
3、初始化 namenode，启动 NameNode
hadoop shell > hadoop namenode -format myhdfs hadoop shell > sh /usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start namenode
4、backup.hadoop 拷贝元数据、启动 NameNode
hadoop shell > hdfs namenode -bootstrapStandby # 从 master.hadoop 拷贝 NameNode 元数据 hadoop shell > sh /usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start namenode
5、master.hadoop、backup.hadoop 启动 zkfc
hadoop shell > sh /usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start zkfc
6、启动 YARN
hadoop shell > sh /usr/local/hadoop-2.8.0/sbin/start-yarn.sh
6、启动 DataNode
hadoop shell > ansible datanode -m shell -a '/usr/local/hadoop-2.8.0/sbin/hadoop-daemon.sh start datanode'
# master.hadoop
8067 NameNode 8376 ResourceManager 8230 DFSZKFailoverController
# backup.hadoop
8255 NameNode 8541 DFSZKFailoverController
# datanode.hadoop
13227 DataNode 13000 NodeManager 12889 JournalNode
七、验证 NameNode HA
# 访问 http://192.168.1.25:50070 显示 active
# 访问 http://192.168.1.26:50070 显示 standby
hadoop shell > kill 8067 # 杀掉 master.hadoop 上的 NameNode hadoop shell > tail -f /usr/local/hadoop-2.8.0/logs/hadoop-hadoop-zkfc-backup.hadoop.log # backup.hadoop 监控 zkfc 日志 2017-05-22 14:08:50,035 INFO org.apache.hadoop.ha.ZKFailoverController: Trying to make NameNode at backup.hadoop/192.168.1.26:8020 active... 2017-05-22 14:08:50,846 INFO org.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNode at backup.hadoop/192.168.1.26:8020 to active state
# 有待进一步完善
