查阅有关官方介绍 http://wiki.apache.org/hadoop/HowToContribute 中有说明：Hadoop本地库只支持*nix平台，已经广泛使用在GNU/Linux平台上，但是不支持 Cygwin  和 Mac OS X 。搜索后发现已经有人给出了Mac OSX 系统下编译生成本地库的patch，下面详细介绍在Mac OSX 平台下编译Hadoop本地库的方法。

[一]、环境说明：

  • Hadoop 2.2.0
  • Mac OS X 10.9.1

详细的环境依赖（protoc、cmake 等）参见：Hadoop2.2.0源码编译 （http://www.micmiu.com/opensource/hadoop/hadoop-build-source-2-2-0/）中介绍。

[二]、Mac OSX 编译本地库的步骤：

 1、checkout Hadoop 2.2.0的源码

1 $svn co https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0/

2、patch 相关补丁

官方讨论地址：https://issues.apache.org/jira/browse/HADOOP-9648 里面有详细介绍

补丁下载链接：https://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch

1 #切换到hadoop 源码的根目录
2 $wget https://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch
3 $patch -p1 < HADOOP-9648.v2.patch

ps：如果要回退patch 执行：patch -RE -p1 < HADOOP-9648.v2.patch 即可。

3、编译本地库

在Hadoop源码的根目录下执行编译本地库命令：

1 $ mvn package -Pdist,native -DskipTests -Dtar

编译成功看到如下日志信息：

1 [INFO] ------------------------------------------------------------------------
2 [INFO] Reactor Summary:
3 [INFO]
4 [INFO] Apache Hadoop Main ................................ SUCCESS [1.511s]
5 [INFO] Apache Hadoop Project POM ......................... SUCCESS [0.493s]
6 [INFO] Apache Hadoop Annotations ......................... SUCCESS [0.823s]
7 [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [0.561s]
8 [INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.245s]
9 [INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [2.465s]
10 [INFO] Apache Hadoop MiniKDC ............................. SUCCESS [0.749s]
11 [INFO] Apache Hadoop Auth ................................ SUCCESS [0.832s]
12 [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.070s]
13 [INFO] Apache Hadoop Common .............................. SUCCESS [1:00.030s]
14 [INFO] Apache Hadoop NFS ................................. SUCCESS [0.285s]
15 [INFO] Apache Hadoop Common Project ...................... SUCCESS [0.049s]
16 [INFO] Apache Hadoop HDFS ................................ SUCCESS [1:13.339s]
17 [INFO] Apache Hadoop HttpFS .............................. SUCCESS [20.259s]
18 [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [0.767s]
19 [INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [0.279s]
20 [INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.046s]
21 [INFO] hadoop-yarn ....................................... SUCCESS [0.239s]
22 [INFO] hadoop-yarn-api ................................... SUCCESS [7.641s]
23 [INFO] hadoop-yarn-common ................................ SUCCESS [5.479s]
24 [INFO] hadoop-yarn-server ................................ SUCCESS [0.114s]
25 [INFO] hadoop-yarn-server-common ......................... SUCCESS [1.743s]
26 [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [6.381s]
27 [INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [0.259s]
28 [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [0.578s]
29 [INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.303s]
30 [INFO] hadoop-yarn-client ................................ SUCCESS [0.233s]
31 [INFO] hadoop-yarn-applications .......................... SUCCESS [0.062s]
32 [INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [0.253s]
33 [INFO] hadoop-mapreduce-client ........................... SUCCESS [0.074s]
34 [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [1.504s]
35 [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [0.242s]
36 [INFO] hadoop-yarn-site .................................. SUCCESS [0.172s]
37 [INFO] hadoop-yarn-project ............................... SUCCESS [1.235s]
38 [INFO] hadoop-mapreduce-client-common .................... SUCCESS [3.664s]
39 [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [0.183s]
40 [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [0.495s]
41 [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [1.296s]
42 [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [0.580s]
43 [INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [0.213s]
44 [INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [0.344s]
45 [INFO] hadoop-mapreduce .................................. SUCCESS [1.303s]
46 [INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [0.257s]
47 [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.925s]
48 [INFO] Apache Hadoop Archives ............................ SUCCESS [0.282s]
49 [INFO] Apache Hadoop Rumen ............................... SUCCESS [0.403s]
50 [INFO] Apache Hadoop Gridmix ............................. SUCCESS [0.283s]
51 [INFO] Apache Hadoop Data Join ........................... SUCCESS [0.197s]
52 [INFO] Apache Hadoop Extras .............................. SUCCESS [0.241s]
53 [INFO] Apache Hadoop Pipes ............................... SUCCESS [8.249s]
54 [INFO] Apache Hadoop OpenStack support ................... SUCCESS [0.492s]
55 [INFO] Apache Hadoop Client .............................. SUCCESS [0.373s]
56 [INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.133s]
57 [INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [0.439s]
58 [INFO] Apache Hadoop Tools Dist .......................... SUCCESS [0.596s]
59 [INFO] Apache Hadoop Tools ............................... SUCCESS [0.044s]
60 [INFO] Apache Hadoop Distribution ........................ SUCCESS [0.194s]
61 [INFO] ------------------------------------------------------------------------
62 [INFO] BUILD SUCCESS
63 [INFO] ------------------------------------------------------------------------
64 [INFO] Total time: 3:44.266s
65 [INFO] Finished at: Fri Jan 17 10:06:17 CST 2014
66 [INFO] Final Memory: 66M/123M
67 [INFO] ------------------------------------------------------------------------
68 micmiu-mbp:trunk micmiu$

编译通过后可在 <HADOOP源码根目录>/hadoop-dist/target/hadoop-2.2.0/lib/ 目录下看到如下内容：

查看源代码
打印帮助
1 micmiu-mbp:lib micmiu$ tree
2 .
3 |____.DS_Store
4 |____native
5 | |____libhadoop.1.0.0.dylib
6 | |____libhadoop.a
7 | |____libhadoop.dylib
8 | |____libhadooppipes.a
9 | |____libhadooputils.a
10 | |____libhdfs.0.0.0.dylib
11 | |____libhdfs.a
12 | |____libhdfs.dylib

然后把 上面生成的本地库 copy到部署环境相应的位置，再建立软连接即可：

1 $ls -s libhadoop.1.0.0.dylib libhadoop.so
2 $ls -s libhdfs.0.0.0.dylib libhdfs.so





错误处理

运行

clean install  package -Pdist -P-cbuild  -DskipTests  -Dtar 

报各种错误

1、报错[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-7:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]
修改hadoop-common-project/hadoop-common/pom.xml 文件中，env.JAVA_HOME改为java.home

2、报错

/hadoop-2.2.0-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression (‘void’ and ‘int’)
[exec] if(setnetgrent(cgroup) == 1) {
[exec] ~~~~~~~~~~~~~~~~~~~ ^ ~
[exec] 1 error generated.
[exec] make[2]: *** [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c.o] Error 1
[exec] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
[exec] make: *** [all] Error 2

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2 -> [Help 1]

修改mvn3的配置文件：/opt/local/share/java/maven3/settings.xml
在<mirrors>…</mirrors>里添加国内源：

<mirrors>
<mirror>
<id>nexus-osc</id>
<mirrorOf>*</mirrorOf>
<name>Nexusosc</name>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>
</mirrors>

在<profiles>…</profiles>标签中增加以下内容：
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7<k>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>

注意修改jdk version number

将刚才的maven 配置文件拷贝到当前用户的home目录下:
settings.xml   copy 到  your_hadoop_usr_home/.m2/
cp settings.xml ~/.m2

3、报错[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Fatal error compiling: Error while executing the compiler. InvocationTargetException: Java heap space

分配内存不足,参考如下为maven配置JVM参数: export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”

4、报错 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Compilation failure
[ERROR] Failure executing javac, but could not parse the error:

执行maven clean，然后再

export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”
三、最重要的一点，build your code是使用这个command line(Only for Mac OS):
mvn clean install -P-cbuild

编译之前， 你在hadoop-2.2.0-src目录（/Users/JuneMAC/hadoop/release-2.2.0）下执行
mvn clean install –DskipTests

上面的成功后，执行下面这个，生成安装包
mvn clean install  package -Pdist -P-cbuild  -DskipTests  -Dtar

执行完成后，可以在/Users/JuneMAC/hadoop/release-2.2.0/hadoop-dist/target/
下找到
hadoop-2.2.0.tar.gz
将上面这个编译好的源码包解压到：
/Users/JuneMAC/hadoop/
然后进行相关配置
解压之后的源码包和官网下载下来的源码包相对比，没有lib目录
相关解释：
“Here we use the additional options to stop compiling the native code.
this is the key reason why we need use -P-cbuild option”
上面这个是原因，好像不是很重要。实际上如果指定-Pdist,native 生成native lib 不成功，查阅有关官方介绍发现：Hadoop本地库只支持*nix平台，已经广泛使用在GNU/Linux平台上，但是不支持 Cygwin  和 Mac OS X 。