Hadoop1.1 和Hadoop 2.4 集群安装版
目录
6.2.1 创建文件夹及将本地文件复制到:hdfs系统中... 40
7.1 hadoop2.2.0 无法连接ResourceManager问题... 42
1 环境准备
1.1 硬件配置
Dell 960 CPU英特尔 酷睿2 四核 Q8300 @ 2.50GHz
内存：4GB
硬盘：320GB
安装Window7,安装虚拟机
1.2 软件
ü CentOS6.5
ü VMware Workstation 10.0.2
ü Secure CRT 7.0
ü WinSCP 5.5.3
ü JDK 1.6.0_43
ü Hadoop1.2.1
ü Eclipse3.6
ü Window7
1.3 网络拓扑结构
192.168.1.53 namenode53
192.168.1.54 datanode54
192.168.1.55 datanode55
192.168.1.56 datanode56
1.4 系统分区
/dev/sda6 4.0G 380M 3.4G 10% /
tmpfs 495M 72K 495M 1% /dev/shm
/dev/sda2 7.9G 419M 7.1G 6% /app
/dev/sda3 7.9G 146M 7.4G 2% /applog
/dev/sda1 194M 30M 155M 16% /boot
/dev/sda5 5.8G 140M 5.3G 3% /data
/dev/sda8 2.0G 129M 1.8G 7% /home
/dev/sda9 2.0G 68M 1.9G 4% /opt
/dev/sda12 2.0G 36M 1.9G 2% /tmp
/dev/sda7 4.0G 3.3G 509M 87% /usr
/dev/sda10 2.0G 397M 1.5G 21% /var
1.5 虚拟机配置
4台虚拟机，配置为：1 CPU*1GBMEM*40GB硬盘 安装CentOS6.5 64位
图1.5-1
分别创建两个用户：root ,hadoop
root 为管理员用户
hadoop 为Hadoop运行用户
1.6 SSH免密码登录
一、创建在用户的home目录下创建 .ssh文件夹
mkdir .ssh
可以隐藏文件夹或文件内容
ls -a
二、 生成证书
证书分为：dsa和rsa
ssh-keygen -t rsa -P '' -b 1024
ssh-keygen 生成命令
-t 表示证书 rsa
-p 密码提示语 ''
-b 证书大小 为：1024
执行后 将会生成密钥文件和私钥文件
ll
-rwx------ 1 apch apache 883 May 20 15:13 id_rsa
-rwx------ 1 apch apache 224 May 20 15:13 id_rsa.pub
三、 把公钥信息写入 authorized_keys 文档中
cat id_rsa.pub >> authorized_keys
（将生成的公钥文件写入 authorized_keys 文件）
四、设置文件和目录权限
设置authorized_keys权限
$ chmod 600 authorized_keys
设置.ssh目录权限
$ chmod 700 -R .ssh
五 修改/etc/ssh/sshd_config (需要使用root用户登录)
vi /etc/ssh/sshd_config
Protocol 2 (仅使用SSH2)
PermitRootLogin yes (允许root用户使用SSH登陆，根据登录账户设置)
ServerKeyBits 1024 (将serverkey的强度改为1024)
PasswordAuthentication no (不允许使用密码方式登陆)
PermitEmptyPasswords no (禁止空密码进行登陆)
RSAAuthentication yes （启用 RSA 认证）
PubkeyAuthentication yes （启用公钥认证）
AuthorizedKeysFile .ssh/authorized_keys
六、重启sshd 服务 (需要使用root用户登录)
service sshd restart
七、本地验证测试
ssh -v localhost (开启登录调试模式)
1.7 JDK安装
从Oracle官网下－jdk-6u43-linux-x64.bin
安装到：/usr/java目录
配置JAVA_HOME:
export JAVA_HOME=/usr/java/jdk1.6.0_43
export JAVA_BIN=/usr/java/jdk1.6.0_43/bin
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH
生效配置文件
source /etc/profile
验证安装：java -version
2 Hadoop1.2.1安装及配置
2.1 下载Hadoop安装介质
下载地址：http://mirrors.hust.edu.cn/apache/hadoop/common/
并上传到Linux系统 rz（前置条件必须安装rz,sz包）
2.2 解压Hadoop安装介质
将hadoop安装到/app/hadoop/的目录下：
tar -zcvf /home/hadoop/hadoop-1.2.1.tar.gz
/app/hadoop/的权限为：hadoop:hadoop（用户、用户组）
2.3 Hadoop配置文件
需要配置修改配置文件如下：
ü hadoop-env.sh
ü core-site.xml
ü hdfs-site.xml
ü mapred-site.xml
ü masters
ü slaves
vi hadoop-env.sh
export JAVA_HOME/usr/java/jdk1.6.0_43
vi core-site.xml
vi hdfs-site.xml
添加以下内容：
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
vi mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>http://192.168.1.53:9001</value>
</property>
</configuration>
vi masters
192.168.1.53
vi slaves
192.168.1.54
192.168.1.55
192.168.1.56
2.4 复制Hadoop安装介质
经过2.3 节配置，将192.168.1.53服务上已配置完成hadoop文件，打包
tar -zcvf hadoop.tar.gz ./hadoop/
再将hadoop.tar.gz 复制到192.168.1.54-56
scp hadoop.tar.gz hadoop@192.168.1.54:/app/hadoop/
在分别登录到192.168.1.54~56 上，
cd /app/hadoop/
tar –zxvf hadoop.tar.gz
2.5 配置HADOOP_HOME
192.68.1.53-56 配置HADOOP_HOME
vi /etc/profile
# set hadoop path
export HADOOP_HOME=/app/hadoop/hadoop
export HADOOP_HOME_WARN_SUPPRESS=1
export PATH=$PATH:$HADOOP_HOME/bin
source /etc/profile
生效配置文件
export HADOOP_HOME_WARN_SUPPRESS=1
解决 启动Hadoop时报了一个警告信息
验证配置：
hadoop version
2.6 格式化 namenode
格式化HDFS文件系统
hadoop namenode -format
2.7 启动Hadoop
使用SecureCRT 7.0登录到192.168.1.53
命令：
cd /app/hadoop/hadoop/bin/
./start-all.sh
查看Hadoop的nameNode进程（192.168.1.53）
说明已启动NameNode,JobTracker,SecondaryNameNode
查看Hadoop的datanode进程：（192.168.1.54）
说明已启动：DataNode,TaskTracker
3 Hadoop1.2.1验证
3.1 Hadoop 控制台
http://192.168.1.53:50070/dfshealth.jsp
http://192.168.1.53:50030/jobtracker.jsp
3.2 Hadoop wordcount运行
hadoop jar hadoop-examples-1.2.1.jar wordcount /tmp/input /tmp/output
通过jobtracker控制台监控结果如下：
4 Hadoop2.4环境准备
4.1 虚拟机准备
从CentOS-53克隆CentOS-57-60
分别虚拟机名称、主机名、IP
CentOS-57 192.168.1.57 namenode57
CentOS-58 192.168.1.58 datanode58
CentOS-59 192.168.1.59 datanode59
CentOS-60 192.168.1.60 datanode60
(会使用VMware虚拟机的的克隆功能，快速的复制已安装好的系统。可是克隆完之后，会发现没有eth0网，切换到root用户下，才能修改，否则权限不足)
配置网卡：
vi /etc/sysconfig/network-scripts/ifcfg-eth0
修改网络卡配置－MAC地址
vi /etc/sysconfig/network-scripts/ifcfg-eth0
修改主机名：
vi /etc/sysconfig/network
重启系统，reboot
4.2 网络拓扑结构
192.168.1.57-60 修改hosts
添加以下内容：
192.168.1.57 namenode57
192.168.1.58 datanode58
192.168.1.59 datanode59
192.168.1.60 datanode60
su –
切换到root用户
vi /etc/hosts
192.168.1.57 namenode57
192.168.1.58 datanode58
192.168.1.59 datanode59
192.168.1.60 datanode60
4.3 系统分区
4.4 虚拟机配置
4.5 SSH免密码登录配置
4.6 JDK安装
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
jdk-7u55-linux-x64.rpm
安装JDK（192.168.1.57-60）
将JDK7安装介质上传到192.168.1.57上“jdk-7u55-linux-x64.rpm”
rz
rpm -i jdk-7u55-linux-x64.rpm
安装成功
cd /usr/java
检查是否安装成功
安装目录：/usr/java/jdk1.7.0_55
配置JAVA_HOME
vi /etc/profile
添加以下内容
export JAVA_HOME=/usr/java/jdk1.7.0_55
export JAVA_BIN=/usr/java/jdk1.7.0_55/bin
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH
生效配置文件
source /etc/profile
检查安装版本：
java -version
表示安装成功
5 Hadoop2.4安装及配置
5.1 下载Hadoop安装介质
http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.0/
并上传到Linux系统 rz命令上传（前置条件必须安装rz,sz包）
登录到192.168.1.57
5.2 解压Hadoop安装介质
安装目录为：/app/hadoop/hadoop2.4
分别在192.168.1.57－60服务器，以hadoop用户登录，并创建目录：hadoop2.4
cd /app/hadoop/
tar -zxvf /home/hadoop/hadoop-2.4.0.tar.gz
注：Hadoop-2.4.0 变化太大了，配置目录都变了
cd /app/hadoop/hadoop-2.4.0/lib/native
file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 32-bit LSB shared object,
环境变量配置：
su - (切换到root下)
vi /etc/profile
添加以下内容：
export HADOOP_HOME=/app/hadoop/hadoop-2.4.0
export HADOOP_HOME_WARN_SUPPRESS=1
export PATH=$PATH:$HADOOP_HOME/bin
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
生效配置文件
source /etc/profile
5.3 编译Hadoop本地库
5.3.1 编译环境准备
基于CentOS6.5 64位操作编译
主要涉及到工具有：hadoop-2.4.0-src.tar.gz、Ant、Maven、JDK、GCC、CMake、openssl
第一步升级系统相关编译所需的软件（升级最新版）：
yum install lzo-devel zlib-devel gcc autoconf automake libtool ncurses-devel openssl-devel
wget http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.0/hadoop-2.4.0-src.tar.gz （源代版）
tar -zxvf hadoop-2.4.0-src.tar.gz
wget http://apache.fayea.com/apache-mirror//ant/binaries/apache-ant-1.9.4-bin.tar.gz
tar -xvf apache-ant-1.9.4-bin.tar.gz
wget http://apache.fayea.com/apache-mirror/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
tar -xvf apache-maven-3.0.5-bin.tar.gz
vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_55
export JAVA_BIN=/usr/java/jdk1.7.0_55/bin
export ANT_HOME=/home/hadoop/ant
export MVN_HOME=/home/hadoop/maven
export FINDBUGS_HOME=/home/hadoop/findbugs-2.0.3
export PATH=$PATH:$JAVA_HOME/bin:$ANT_HOME/bin:$MVN_HOME/bin:$FINDBUGS_HOME/bin
生产配置文件：
source /etc/profile
验证是否配置成功
ant –version
mvn -version
findbugs –version
验证结果：
安装protobuf（以root用户登录）
wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
tar zxf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure
make
make install
protoc --version
安装cmake（以root用户登录）
wget http://www.cmake.org/files/v2.8/cmake-2.8.12.2-Linux-i386.tar.gz
./bootstrap
make
make install
cmake –version
为了加速编译，将maven镜像库指向：开源中国
cd maven/conf
vi settings.xml
添加以下内容：
<mirror>
<id>nexus-osc</id>
<mirrorOf>*</mirrorOf>
<name>Nexus osc</name>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>
<mirror>
<id>nexus-osc-thirdparty</id>
<mirrorOf>thirdparty</mirrorOf>
<name>Nexus osc thirdparty</name>
<url>http://maven.oschina.net/content/repositories/thirdparty/</url>
</mirror>
<profile>
<id>jdk-1.4</id>
<activation>
<jdk>1.4</jdk>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
详细说明可参见：
http://maven.oschina.net/help.html
5.3.2 编译Hadoop
mvn package -DskipTests -Pdist,native –Dtar
此时在下载maven依赖所有包及插件
慢慢等待中……（花6个小时，终于看到一编译错误）
编译成功，检查nativelib 是否编译成功
cd hadoop-dist/target/hadoop-2.4.0/lib/native
file libhadoop.so.1.0.0
libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
代表编译成功
错误1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 46.796s
[INFO] Finished at: Wed Jun 04 13:28:37 CST 2014
[INFO] Final Memory: 36M/88M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hadoop-common: Could not resolve dependencies for project org.apache.hadoop:hadoop-common:jar:2.4.0: Failure to find org.apache.commons:commons-compress:jar:1.4.1 in https://repository.apache.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots.https has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-common
解决方法：
根据上面日志提示说找不到“org.apache.commons:commons-compress:jar:1.4.1”,
直接将本地（Windows）包复制到Linux系统中，解决了。
错误2
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:16.693s
[INFO] Finished at: Wed Jun 04 13:56:31 CST 2014
[INFO] Final Memory: 48M/239M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native"): error=2, 没有那个文件或目录
[ERROR] around Ant part ...<exec dir="/home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:133 in /home/hadoop/hadoop-2.4.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-common
解决方法：
是没有安装cmake导致的，再重新安装cmake；参考《5.3.1编译环境准备》
错误3
错误提示是找不到相应的文件和不能创建目录，在网上没有相关错误（根据自己经验修改目录权限为：775，让目录有创建文件或文件夹的权限，另外最好保证hadoop编译目录有2.5G至4G的空间）
chmod -Rf 775 ./ hadoop-2.4.0-src
main:
[mkdir] Created dir: /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-pipes ---
[INFO] Executing tasks
错误3
main:
[mkdir] Created dir: /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native
[exec] -- The C compiler identification is GNU 4.4.7
[exec] -- The CXX compiler identification is GNU 4.4.7
[exec] -- Check for working C compiler: /usr/bin/cc
[exec] -- Check for working C compiler: /usr/bin/cc -- works
[exec] -- Detecting C compiler ABI info
[exec] -- Detecting C compiler ABI info - done
[exec] -- Check for working CXX compiler: /usr/bin/c++
[exec] -- Check for working CXX compiler: /usr/bin/c++ -- works
[exec] -- Detecting CXX compiler ABI info
[exec] -- Detecting CXX compiler ABI info - done
[exec] CMake Error at /usr/local/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:108 (message):
[exec] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_LIBRARIES
[exec] OPENSSL_INCLUDE_DIR)
[exec] Call Stack (most recent call first):
[exec] /usr/local/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE)
[exec] /usr/local/share/cmake-2.8/Modules/FindOpenSSL.cmake:313 (find_package_handle_standard_args)
[exec] CMakeLists.txt:20 (find_package)
[exec]
[exec]
[exec] -- Configuring incomplete, errors occurred!
[exec] See also "/data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native/CMakeFiles/CMakeOutput.log".
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [13.745s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [5.538s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [7.296s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.568s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [5.858s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [8.541s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [8.337s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [7.348s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [4.926s]
[INFO] Apache Hadoop Common .............................. SUCCESS [2:35.956s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [18.680s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.059s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [5:03.525s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [38.335s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [23.780s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [8.769s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.159s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.134s]
[INFO] hadoop-yarn-api ................................... SUCCESS [2:07.657s]
[INFO] hadoop-yarn-common ................................ SUCCESS [1:10.680s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.165s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [24.174s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [27.293s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [5.177s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [11.399s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [28.384s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.346s]
[INFO] hadoop-yarn-client ................................ SUCCESS [12.937s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.108s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [5.303s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.212s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.050s]
[INFO] hadoop-yarn-project ............................... SUCCESS [8.638s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.135s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [43.622s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [36.329s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [6.058s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [20.058s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [16.493s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [11.685s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [3.222s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [12.656s]
[INFO] hadoop-mapreduce .................................. SUCCESS [8.060s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [8.994s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [15.886s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [6.659s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [15.722s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [11.778s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [5.953s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [6.414s]
[INFO] Apache Hadoop Pipes ............................... FAILURE [3.746s]
[INFO] Apache Hadoop OpenStack support ................... SKIPPED
[INFO] Apache Hadoop Client .............................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED
[INFO] Apache Hadoop Tools Dist .......................... SKIPPED
[INFO] Apache Hadoop Tools ............................... SKIPPED
[INFO] Apache Hadoop Distribution ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19:43.155s
[INFO] Finished at: Wed Jun 04 17:40:17 CST 2014
[INFO] Final Memory: 79M/239M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec dir="/data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/native" executable="cmake" failonerror="true">... @ 5:123 in /data/hadoop/hadoop-2.4.0-src/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
根据网上提示（ 下面需要再安装openssl-devel,安装命令yum install openssl-devel,此步不做的话会报如下错误
[exec] CMake Error at /usr/share/cmake/Modules/FindOpenSSL.cmake:66 (MESSAGE):
[exec] Could NOT find OpenSSL
[exec] Call Stack (most recent call first):
[exec] CMakeLists.txt:20 (find_package)
[exec]
[exec]
[exec] -- Configuring incomplete, errors occurred!
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluen ... oExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-pipes
）
错误连接：http://f.dataguru.cn/thread-189176-1-1.html
原因是：在安装openssl-devel,少写一个l,重新安装一下
解决方法：重新安装openssl-devel
yum install openssl-devel
5.3.3 编译总结
1、 必须安装（yum install lzo-devel zlib-devel gcc autoconf automake libtool ncurses-devel openssl-devel）
2、 必须安装（protobuf，CMake）编译工具
3、 必须配置（ANT、MAVEN、FindBugs）
4、 将maven库指向开源中国，这样就可以加快编译速度，即加快下载依赖jar包速度
5、 编译出错需求详细观察出错日志，根据错误日志分析原因再结束百度和Google解决错误；
5.4 Hadoop 配置文件
cd hadoop-2.4.0
cd etc/hadoop
core-site.xml
yarn-site.xml
hdfs-site.xml
mapred-site.xml
hadoop-env.sh
vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://namenode57:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/current/tmp</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/app/hadoop/current/data</value>
</property>
</configuration>
vi yarn-site.xml
<property>
<name>yarn.resourcemanager.address</name>
<value>namenode57:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>namenode57:18030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>namenode57:18088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>namenode57:18025</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>namenode57:18141</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
vi hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.rpc-address</name>
<value>namenode57:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/app/hadoop/current/dfs</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/app/hadoop/current/data</value>
</property>
</configuration>
vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
vi hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_55
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
5.5 复制Hadoop安装介质
scp hadoop-2.4.0.tar.gz hadoop@192.168.1.58:/app/hadoop/
分别复制到：192.168.1.58-60
5.6 格式化NameNode
./hdfs namenode -format
5.7 启动Hadoop
cd /app/hadoop/hadoop-2.4.0/
./start-all.sh
查看进程hadoop进程：
jps
ps –ef|grep java
14/06/04 07:48:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on
启动时一直报错，准备编译Hadoop64位依赖包（5.3编译Hadoop本地库），再进行替换
替换掉32位的native库
删除原32位的native库
cd /app/hadoop/hadoop-2.4.0/lib
rm -rf native/
将5.3节编译好native 64位的库复制到：/app/hadoop/hadoop-2.4.0/lib
cd /data/hadoop/hadoop-2.4.0-src/hadoop-dist/target/hadoop-2.4.0/lib
cp -r ./native /app/hadoop/hadoop-2.4.0/lib/
错误1：
2014-06-04 18:30:57,450 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
java.lang.IllegalArgumentException: The ServiceName: mapreduce.shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:98)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:220)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:186)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:357)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:404)
2014-06-04 18:30:57,458 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
解决方法：
vi /app/hadoop/hadoop-2.4.0/etc/hadoop/yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
改为
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
总结：
从2.0以后启动和停止hadoop的命令start-all.sh和stop-all.sh不建议使用，开始摒弃掉取而代之的将使用start-dfs.sh和start-yarn.sh启动hadoop,详细请看官方说明。
5.8 检查Hadoop启动情况
5.8.1 检查namenode是否启动
在192.168.1.57检查
jps
5.8.2 编译Hadoop
分别检查192.168.1.58~60
Jps
6 Hadoop2.4验证
6.1 Hadoop 控制台
HDFS
http://192.168.1.57:50070/dfshealth.html#tab-overview
http://namenode57:8088/cluster/
6.2 Hadoop wordcount运行
6.2.1 创建文件夹及将本地文件复制到:hdfs系统中
1、创建/tmp/input文件夹
hadoop fs -mkdir /tmp
hadoop fs -mkdir /tmp/input
2、将本地文件复制到hdfs系统中
hadoop fs -put /usr/hadoop/file* /tmp/input
3、查看test.txt文件是否成功上传到hdfs上
hadoop fs -ls /tmp/input
6.2.1 运行wordcount程序
hadoop jar hadoop-mapreduce-examples-2.4.0.jar wordcount /tmp/input /tmp/output
7 Hadoop安装错误
7.1 hadoop2.4无法连接ResourceManager问题
错误日志：
ountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-06-05 07:01:07,154 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-06-05 07:01:08,156 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-06-05 07:01:09,159 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-06-05 07:01:10,161 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-06-05 07:01:11,164 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-06-05 07:01:12,166 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-06-05 07:01:13,169 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2014-06-05 07:01:14,171 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
解决方法，在yarn.xml文件里配置，所有机器都修改
<property>
<name>yarn.resourcemanager.hostname</name>
<value>namenode57</value>
</property>
