[置顶] 单机版hadoop实例安装

目标：运行单机版hadoop

http://localhost:50030mapredule监控界面

http://localhost:50070HDFS监控页面

-->安装linux系统

-->java环境

-->ssh免密码登陆-->防火墙iptables

-->配置hadoop

环境准备：Centos 6.3  http://mirror.symnds.com/distributions/CentOS-vault/6.3/isos/i386/   CentOS-6.3-i386-bin-DVD1.iso

JDK http://download.oracle.com/otn-pub/java/jdk/7u45-b18/jdk-7u45-linux-i586.tar.gz?AuthParam=1384763306_45788258a165f5f772a28d5a5fb88c7d jdk-7u45-linux-i586.gz

hadoop http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz hadoop-2.2.0.tar.gz

安装配置jDK

[ge@localhost ~]$ tar -zxvf jdk-7u45-linux-i586.gz-----解压

[ge@localhost ~]$ tar -zxvf hadoop-2.2.0.tar.gz

[ge@localhost ~]$ su  root

[root@localhost ge]# mv jdk1.7.0_45/ /usr/java1.7

[root@localhost ge]# mv hadoop-2.2.0 /usr/hadoop

[root@localhost ge]# vim /etc/profile

添加以下行：

 

export JAVA_HOME=/usr/java1.7
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

执行以下命令是文件马上生效。

 

[root@localhost ge]# source /etc/profile

测试java版本：

[root@localhost ge]# java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) Client VM (build 24.45-b08, mixed mode)

ssh免秘钥登陆：

[root@localhost ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

[root@localhost ~]# ll ~/.ssh/
总用量 12
-rw-------. 1 root root 1675 11月 19 01:29 id_rsa
-rw-r--r--. 1 root root  408 11月 19 01:29 id_rsa.pub
-rw-r--r--. 1 root root  391 11月 19 01:16 known_hosts

[root@localhost ~]# cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys
[root@localhost ~]# ssh localhost
Last login: Tue Nov 19 01:16:10 2013 from localhost

配置hadoop:

编辑/usr/hadoop/etc/hadoop/hadoop-env.sh
替换exportJAVA_HOME=${JAVA_HOME}为如下：exportJAVA_HOME=/usr/java1.7

编辑/usr/hadoop/etc/hadoop/core-site.xml，在<configuration>中添加如下：

<property>
  <name>hadoop.tmp.dir</name>
 <value>/usr/hadoop/tmp/hadoop-${user.name}</value>
 <description>A base for other temporarydirectories.</description>
</property>

<property>
 <name>fs.default.name</name>
 <value>hdfs://localhost:8010</value>
 <description>The name of the default file system.  A URI whose
 scheme and authority determine the FileSystem implementation.  The
 uri’s scheme determines the config property (fs.SCHEME.impl) naming
 the FileSystem implementation class. The uri’s authority is used to
 determine the host, port, etc. for a filesystem.</description>
</property>

备注：配置了/usr/hadoop/tmp/这个目录，必须执行mkdir /usr/hadoop/tmp/创建它，否则后面运行会报错。

编辑/usr/hadoop/etc/hadoop/mapred-site.xml：
(1)   mv /usr/hadoop/etc/hadoop/mapred-site.xml.template /usr/hadoop/etc/hadoop/mapred-site.xml
(2)   在<configuration>中添加如下:

<property>
 <name>mapred.job.tracker</name>
 <value>localhost:54311</value>
 <description>The host and port that the MapReduce job tracker runs
 at.  If “local”, thenjobs are run in-process as a single map
 and reduce task.
  </description>
</property>

<property>
 <name>mapred.map.tasks</name>
 <value>10</value>
 <description>As a rule of thumb, use 10x the number of slaves(i.e., number of tasktrackers).
  </description>
</property>

<property>
 <name>mapred.reduce.tasks</name>
 <value>2</value>
 <description>As a rule of thumb, use 2x the number of slaveprocessors (i.e., number of tasktrackers).
  </description>
</property>

编辑/usr/hadoop/etc/hadoop/hdfs-site.xml，在<configuration>中添加如下：

<property>
 <name>dfs.replication</name>
 <value>1</value>
 <description>Default block replication.
 The actual number of replications can be specified when the file iscreated.
 The default is used if replication is not specified in create time.
  </description>
</property>

运行Hadoop
在初次运行Hadoop的时候需要初始化Hadoop文件系统，命令如下：
$cd /usr/hadoop/bin
$./hdfs namenode -format
如果执行成功，你会在日志中(倒数几行)找到如下成功的提示信息：
common.Storage: Storage directory/usr/hadoop/tmp/hadoop-hduser/dfs/name has been successfully formatted.
运行命令如下：
$cd /usr/hadoop/sbin/
$./start-dfs.sh
注：该过程需要多次输入密码, 如果不想多次输入密码，可先用ssh建立信任。
$ jps
4266 SecondaryNameNode
4116 DataNode
4002 NameNode
注：用jps查看启动了三个进程。

$./start-yarn.sh
cd /usr/hadoop/sbin/
$ jps
4688 NodeManager
4266 SecondaryNameNode
4116 DataNode
4002 NameNode
4413 ResourceManager

查看Hadoop资源管理器
http://192.168.128.129:8088/，将其中的192.168.128.129替换为你的实际IP地址。

测试Hadoop
cd /usr
$ wget http://www.gutenberg.org/cache/epub/20417/pg20417.txt
$ cd hadoop
$ bin/hdfs dfs -mkdir /tmp
$ bin/hdfs dfs -copyFromLocal /usr/pg20417.txt /tmp 
$ bin/hdfs dfs -ls /tmp
$ bin/hadoop jar./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /tmp/ /tmp-output

如果一切正常的话，会输入相应的结果，可以从屏幕输出看到。
$bin/hadoop fs -ls /tmp-output可以查看/tmp-output任务完成情况，会显示两个文件：
-rw-r–r–   1 hadoop supergroup          0 2013-10-28 23:09 /tmp-output/_SUCCESS
-rw-r–r–   1 hadoop supergroup     196192 2013-10-28 23:09 /tmp-output/part-r-00000
通过 bin/hadoop fs -cat /tmp-output/part-r-00000 查看结果

停止Hadoop
若停止hadoop，依次运行如下命令：
$./stop-yarn.sh
$./stop-dfs.sh