hadoop hdfs简单集群安装
hadoop集群三个节点
hadoop-master1, hadoop-slave1,hadoop-slave2
1、安装jdk 1.8
2、免密码登录,配置hosts解析 所有机器上都要配置一致,包括自身host解析
3、下载hadoop2.9 解压到 /usr/local/
4、配置环境变量
#java环境变量 export JAVA_HOME=/usr/local/jdk1.8.0_131 export JRE_HOME=/usr/local/jdk1.8.0_131/jre export PATH=$PATH:/usr/local/jdk1.8.0_131/bin:/usr/local/jdk1.8.0_131/jre/bin #hadoop环境变量 export HADOOP_HOME=/usr/local/hadoop export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin export HADOOP_INSTALL=$HADOOP_HOME
5、编辑hadoop配置文件 /usr/local/hadoop/etc/hadoop/ 目录下
hadoop-env.sh, core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml , masters, slaves
hadoop-enf.sh
#加入JAVA_HOME路径 export JAVA_HOME=/usr/local/jdk1.8.0_131
core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop-master1:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/opt/hadoop/tmp</value> <description>Abase for other temporary directories.</description> </property> </configuration>
hdfs-site.xml
<configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>master:9001</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/opt/hadoop/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/opt/hadoop/dfs/data</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> </configuration>
mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
yarn-site.xml
<configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
masters
hadoop-master1
slaves
hadoop-salve1
hadoop-slave2
6、hadoop可执行文件 /usr/local/hadoop/[bin,sbin]
因为hadoop路径已经加入环境变量
格式化 hadoop namenode -format
启动 start-all.sh
关闭stop-all.sh
查看是否启动master和slaves上分别执行 jps
查看hdfs集群 hdfs dfsadmin -report
7、常见问题:保持 slave上datanode(目录)/current/VERSION 与 master上namenode(目录)/current/VERSION中
slave datanode 与master namenode 保持一致 clusterID=CID-3d8485ee-025a-4ef2-8533-7faa94cf8d5a
8、 所有master,slave中hadoop配置文件完全相同,只需要配置一遍然后拷贝到其他机器中即可
存放namenode 和datanode的路径需要指定(hdfs-site.xml),并手动创建目录