hadoop 2.3 最新版本启动 验证通过

           

192.168.81.132 -> hadoop1 namenode:   

192.168.81.130 ->  hadoop2 datanode1:         

192.168.81.129 -> hadoop3 datanode2;

192.168.81.131 -> hadoop4 datanode3;

一、创建账号

1.所有节点创建用户   

useradd hadoop   

passwd hadoop

2.所有节点创建目录  

mkdir -p /home/hadoop/source  

mkdir -p /home/hadoop/tools

3.Slave节点创建目录  

mkdir -p /hadoop/hdfs  

mkdir -p /hadoop/tmp  

mkdir -p /hadoop/log  

chmod -R 777 /hadoop

二、修改主机名

所有节点修改

1.vim /etc/sysconfig/network ，

修改 HOSTNAME=hadoopx

2.vim /etc/hosts

192.168.81.132   hadoop1

192.168.81.130   hadoop2

192.168.81.129   hadoop3

192.168.81.131   hadoop4

3.执行 hostname hadoopx

4.重新登录，即可

三、免密码登录

注意：非root用户免密码登录，需要执行 chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys 如果不修改权限，非root用户无法免密码登录

四、安装JDK(略)

五、配置环境变量  

1. /etc/profile

export HADOOP_HOME=/home/hadoop/hadoop

export PATH=$PATH:$HADOOP_HOME/bin

export PATH=$PATH:$HADOOP_HOME/sbin

export HADOOP_MAPARED_HOME=${HADOOP_HOME}

export HADOOP_COMMON_HOME=${HADOOP_HOME}

export HADOOP_HDFS_HOME=${HADOOP_HOME}

export YARN_HOME=${HADOOP_HOME}

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

export HDFS_CONF_DIR=${HADOOP_HOME}/etc/hadoop export

YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop  

2. hadoop-env.sh  

在末尾添加 export JAVA_HOME=/usr/java/jdk1.6.0_27

六、Hadoop 2.3安装  

1.core-site.xml

在 configuration 节点 里面添加属性

 <property>  

<name>hadoop.tmp.dir</name>  

<value>/hadoop/tmp</value>  

<description>A base for other temporary directories.</description>  

</property>  

<property>

 <name>fs.default.name</name>

 <value>hdfs://192.168.81.132:9000</value>  

</property>  

添加 httpfs 的 选项

<property>  

<name>hadoop.proxyuser.root.hosts</name>  

<value>192.168.1.201</value>  

</property>  

<property>  

<name>hadoop.proxyuser.root.groups</name>

 <value>*</value>  

</property>

2.添加slaves节点   192.168.81.129   192.168.81.130   192.168.81.131

3.配置hdfs-site.xml

/etc/hadoop/hdfs-site.xml 添加 节点

<property>  

<name>dfs.replication</name>

 <value>3</value>

</property>

<property>

 <name>dfs.namenode.name.dir</name>

 <value>file:/hadoop/hdfs/name</value>

 <final>true</final>

</property>

<property>  

<name>dfs.federation.nameservice.id</name>  

<value>ns1</value>

</property>

<property>  

<name>dfs.namenode.backup.address.ns1</name>

 <value>192.168.81.132:50100</value>

</property>

<property>

 <name>dfs.namenode.backup.http-address.ns1</name>

 <value>192.168.81.132:50105</value>

</property>

<property>

 <name>dfs.federation.nameservices</name>  

<value>ns1</value>

</property>

<property>

 <name>dfs.namenode.rpc-address.ns1</name>

 <value>192.168.81.132:9000</value>

</property>

<property>

 <name>dfs.namenode.rpc-address.ns2</name>  

<value>192.168.81.132:9000</value>

</property>

<property>  

<name>dfs.namenode.http-address.ns1</name>

 <value>192.168.81.132:23001</value>

</property>

<property>  

<name>dfs.namenode.http-address.ns2</name>  

<value>192.168.81.132:13001</value>

</property>

<property>

 <name>dfs.dataname.data.dir</name>

 <value>file:/hadoop/hdfs/data</value>

 <final>true</final>

</property>

<property>  

<name>dfs.namenode.secondary.http-address.ns1</name>

 <value>192.168.81.132:23002</value>

</property>

<property>

 <name>dfs.namenode.secondary.http-address.ns2</name>

 <value>192.168.81.132:23002</value>

</property>

<property>

 <name>dfs.namenode.secondary.http-address.ns1</name>

 <value>192.168.81.132:23003</value>

</property>

<property>

 <name>dfs.namenode.secondary.http-address.ns2</name>  

<value>192.168.81.132:23003</value>

</property>

4.配置yarn-site.xml 添加 节点

<property>  <name>yarn.resourcemanager.address</name>  <value>192.168.81.132:18040</value> </property>

<property>  <name>yarn.resourcemanager.scheduler.address</name>  <value>192.168.81.132:18030</value> </property>

<property>  <name>yarn.resourcemanager.webapp.address</name>  <value>192.168.81.132:18088</value> </property>

<property>  <name>yarn.resourcemanager.resource-tracker.address</name>  <value>192.168.81.132:18025</value> </property>

<property>  <name>yarn.resourcemanager.admin.address</name>  <value>192.168.81.132:18141</value> </property>

<property>  <name>yarn.nodemanager.aux-services</name>  <value>mapreduce.shuffle</value> </property> 5.配置httpfs-site.xml 略

七、配置DataNode  

1.复制hadoop-2.3至各Data节点  

2.复制环境变量/etc/profile，生效

八、启动并验证

1.格式化HDFS   hadoop namenode -format -clusterid clustername

2.启动HDFS   start-dfs.sh

3.启动任务管理器   start-yarn.sh

4.启动httpfs   httpfs.sh start

5.NameNode  验证进程

 NameNode  Bootstrap  SecondaryNameNode  ResourceManager

6.DataNode 验证进程  DataNode  NodeManager

7.测试HDFS读写，JOB的运行

 

此过程本人亲自一步步总结并验证，有问题可以email:bobsoft@foxmail.com