hadoop ——完全分布式环境搭建
hadoop 完全分布式环境搭建
1、虚拟机角色分配:
192.168.44.184 hadoop02 NameNode/DataNode ResourceManager/NodeManager
192.168.44.185 hadoop03 DataNode NodeManager
192.168.44.186 hadoop04 DataNode NodeManager
每台机器配置好hostname,hosts:/etc/hostname 中配置好hostname
/etc/hosts 中配置好hosts:
192.168.44.184 hadoop02
192.168.44.185 hadoop03
192.168.44.186 hadoop04
2、sskey免登录设置
1)ssh-keygen -t rsa (会在.ssh目录生成公钥和私钥,每台机器都要生成)
2)以hadoop02为主,将公钥copy到各个机器
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop02
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop03
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop04
3、jdk的安装(在hadoop02上配置,另外两台机器稍后复制过去)
1) 直接下载jdk包放到安装目录即可(jdk-8u131-linux-x64.tar.gz解压到安装目录 tar zxvf ./jdk-8u131-linux-x64.tar.gz -C /home/hadoop/app/)
2) 配置环境变量 直接在 .bash_profile配置
export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131
export PATH=$JAVA_HOME/bin:$PATH
4.下载安装hadoop-2.6.0-cdh5.7.0(在hadoop02上配置,另外两台机器稍后复制过去)
cdh下载地址:(http://archive.cloudera.com/cdh5/cdh/5/)
1)直接下载hadoop-2.6.0-cdh5.7.0.tar.gz包放到安装目录即可
2)配置环境变量
export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$PATH
5、配置修改hadoop相关配置文件(在hadoop02上配置,另外两台机器稍后复制过去)
1)hadoop_home/etc/hadoop/hadoop-env.sh 配置jdk
export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131
2)etc/hadoop/core-site.xml
<property> <name>fs.defaultFS</name> <value>hdfs://hadoop02:8020</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/app/tmp</value> </property>
3)etc/hadoop/hdfs-site.xml 设置副本系数和name、data的存放路径
<property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/home/hadoop/app/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/home/hadoop/app/tmp/dfs/data</value> </property>
4)etc/hadoop/yarn-site.xml
<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>hadoop02</value> </property>
5)etc/hadoop/mapred-site.xml
<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>
6)/etc/hadoop/slaves(标明从节点)
hadoop02
hadoop03
hadoop04
6、分发安装包到 hadoop03、hadoop03节点上
scp -r ~/app hadoop@hadoop03:~/
scp -r ~/app hadoop@hadoop04:~/
scp ~/.bash_profile hadoop@hadoop03:~/
scp ~/.bash_profile hadoop@hadoop04:~/
在hadoop03,hadoop04上分别source下.bash_profile文件
7、NameNode格式化:只要在hadoop02上格式化
bin/hdfs namenode -format
8、启动hadoop集群。在NameNode启动
sbin/start-all.sh
启动日志:
[hadoop@hadoop02 hadoop-2.6.0-cdh5.7.0]$ ./sbin/start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
18/08/19 23:20:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [hadoop02]
hadoop02: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop02.out
hadoop02: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop02.out
hadoop04: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop04.out
hadoop03: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop03.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 78:05:94:d6:0b:e2:78:7d:93:3c:90:d0:ab:7c:ed:e2.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop02.out
18/08/19 23:21:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop02.out
hadoop03: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop03.out
hadoop04: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop04.out
hadoop02: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop02.out
9、验证
1)jps验证:
[hadoop@hadoop02 hadoop-2.6.0-cdh5.7.0]$ jps
3862 SecondaryNameNode
4086 NodeManager
3591 NameNode
3991 ResourceManager
3678 DataNode
4399 Jps
[hadoop@hadoop03 ~]$ jps
3460 NodeManager
3576 Jps
3372 DataNode
[hadoop@hadoop04 ~]$ jps
3284 NodeManager
3400 Jps
3196 DataNode
2)web页面验证:http://hadoop02:50070 (hdfs)
http://hadoop02:8088 (yarn)
10、停止
sbin/stop-all.sh