64位的linux装的hadoop是32位的,需要手工编译-解决方案(摘抄)

64位的linux装的hadoop是32位的,需要手工编译。

遇到的问题描述:
[root@db96 hadoop]# hadoop dfs -put ./in
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

14/07/17 17:07:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
put: `./in': No such file or directory
原因查找:
查看本地文件:
[root@db96 hadoop]# file /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 
/usr/local/hadoop/lib/native/libhadoop.so.1.0.0: ELF 32-bit LSB shared object, 
 Intel 80386, version 1 (SYSV), dynamically linked, not stripped
 是32位的hadoop,安装在了64位的linux系统上。lib包编译环境不一样,所以不能使用。
 悲剧了,装好的集群没法用。
 
 解决办法:重新编译hadoop.//就是重新编译hadoop软件。 (本例文是在从库db99上编译。你也可以在master db96上编译//只要机器的环境一致。)
 
已编译好的hadoop-2.7.1 : http://pan.baidu.com/s/1o6J1Z0A (百度网盘分享)
 
编译步骤:
 下载程序代码:(linux下安装svn)
 # yum install svn
 [root@db99 data]# /usr/local/svn/bin/svn checkout 'http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0'
 [root@db99 release-2.2.0]# ll
总用量 84
-rw-r--r--  1 root root  9968 7月  17 18:51 BUILDING.txt
drwxr-xr-x  3 root root  4096 7月  17 18:51 dev-support
drwxr-xr-x  4 root root  4096 7月  17 18:51 hadoop-assemblies
drwxr-xr-x  3 root root  4096 7月  17 18:51 hadoop-client
drwxr-xr-x  9 root root  4096 7月  17 18:51 hadoop-common-project
drwxr-xr-x  3 root root  4096 7月  17 18:50 hadoop-dist
drwxr-xr-x  7 root root  4096 7月  17 18:51 hadoop-hdfs-project
drwxr-xr-x 11 root root  4096 7月  17 18:51 hadoop-mapreduce-project
drwxr-xr-x  4 root root  4096 7月  17 18:51 hadoop-maven-plugins
drwxr-xr-x  3 root root  4096 7月  17 18:50 hadoop-minicluster
drwxr-xr-x  4 root root  4096 7月  17 18:50 hadoop-project
drwxr-xr-x  3 root root  4096 7月  17 18:50 hadoop-project-dist
drwxr-xr-x 12 root root  4096 7月  17 18:50 hadoop-tools
drwxr-xr-x  4 root root  4096 7月  17 18:51 hadoop-yarn-project
-rw-r--r--  1 root root 16569 7月  17 18:51 pom.xml


安装开发环境
1. 安装必要的包:
[root@db99 data]#
yum install autoconfautomake libtool cmake ncurses-devel openssl-devel gcc* --nogpgcheck
出现下面的字样就说明安装成功。
 
作为依赖被安装:
  cloog-ppl.x86_64 0:0.15.7-1.2.el6                                            
  cpp.x86_64 0:4.4.7-16.el6                                                    
  ecj.x86_64 1:3.4.2-6.el6                                                     
  java-1.5.0-gcj.x86_64 0:1.5.0.0-29.1.el6                                     
  java_cup.x86_64 1:0.10k-5.el6                                                
  keyutils-libs-devel.x86_64 0:1.4-5.el6                                       
  krb5-devel.x86_64 0:1.10.3-42.el6                                            
  libcom_err-devel.x86_64 0:1.41.12-22.el6                                     
  libgcj.x86_64 0:4.4.7-16.el6                                                 
  libgcj-devel.x86_64 0:4.4.7-16.el6                                           
  libgfortran.x86_64 0:4.4.7-16.el6                                            
  libgnat.x86_64 0:4.4.7-16.el6                                                
  libgnat-devel.x86_64 0:4.4.7-16.el6                                          
  libobjc.x86_64 0:4.4.7-16.el6                                                
  libselinux-devel.x86_64 0:2.0.94-5.8.el6                                     
  libsepol-devel.x86_64 0:2.0.41-4.el6                                         
  libstdc++-devel.x86_64 0:4.4.7-16.el6                                        
  mpfr.x86_64 0:2.4.1-6.el6                                                    
  ppl.x86_64 0:0.10.2-11.el6                                                   
  sinjdoc.x86_64 0:0.5-9.1.el6                                                 

完毕!

2. 安装maven,下载并解压。
http://maven.apache.org/download.cgi  //下载最新的压缩包
http://pan.baidu.com/s/1i3KYdXb (百度云盘分享)可下载 apache-maven-3.3.3-bin.tar 版本
 
[root@db99 ~]# tar -xvf apache-maven-3.3.3-bin.tar
[root@db99 ~]# ln -s /usr/hadoop/apache-maven-3.3.3/ /usr/hadoop/maven
[root@db99 local]# vim /etc/profile  //添加环境变量中
export MAVEN_HOME=/usr/hadoop/maven
export PATH=$MAVEN_HOME/bin:$PATH 
 
将解压后的文件夹中的所有文件复制到目录 /usr/hadoop/maven 下
 
3. 安装protobuf
http://pan.baidu.com/s/1pJ3kXtD (百度云盘分享)
下载:protobuf-2.5.0.tar.gz  并解压
[root@db99 protobuf-2.5.0]# pwd
/usr/hadoop/protoc
[root@db99 protobuf-2.5.0]# ./configure --prefix=/usr/hadoop/protoc/
[root@db99 protobuf-2.5.0]# make
[root@db99 protobuf-2.5.0]# make check
[root@db99 protobuf-2.5.0]# make install
[root@db99 protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
安装成功。
添加环境变量:
vi /etc/profile
export JAVA_HOME=/usr/java/latest
export PATH=.:/usr/hadoop/protoc/bin:$JAVA_HOME/bin:$PATH
 
关键点:如果解压后的文件是在一个文件夹中,则需要将文件夹中的所有文件复制到安装目录下

4. 编译hadoop
[root@db99 release-2.2.0]# pwd
/data/release-2.2.0
[root@db99 release-2.2.0]# ls
BUILDING.txt       hadoop-common-project     hadoop-maven-plugins  hadoop-tools
dev-support        hadoop-dist               hadoop-minicluster    hadoop-yarn-project
hadoop-assemblies  hadoop-hdfs-project       hadoop-project        pom.xml
hadoop-client      hadoop-mapreduce-project  hadoop-project-dist
 
[root@db99 release-2.2.0]# mvn package -Pdist,native -DskipTests -Dtar
..............编译需要较长时间大概1个小时左右。
如果出现如下错误:
 
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-auth: Compilation failure: Compilation failure:
[ERROR] /home/hduser/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] error: cannot access AbstractLifeCycle
[ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
[ERROR] /home/hduser/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,29] error: cannot access LifeCycle
[ERROR] class file for org.mortbay.component.LifeCycle not found
[ERROR] /home/hduser/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[98,10] error: cannot find symbol
[ERROR] symbol:   method start()
[ERROR] location: variable server of type Server
[ERROR] /home/hduser/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[104,12] error: cannot find symbol
[ERROR] -> [Help 1]
 
需要修改源码下边的hadoop-common-project/hadoop-auth/pom.xml
vi ~/hadoop-common-project/hadoop-auth/pom.xml
[root@db99 release-2.2.0]# vim /data/release-2.2.0/hadoop-common-project/hadoop-auth/pom.xml 
在第55行下添加:
 56     <dependency>
 57         <groupId>org.mortbay.jetty</groupId>
 58         <artifactId>jetty-util</artifactId>
 59         <scope>test</scope>                                                                          
 60     </dependency>
 
  1. <plugin>  
  2.     <groupId>org.mortbay.jetty</groupId>  
  3.     <artifactId>maven-jetty-plugin</artifactId>  
  4.     <version>6.0.0</version>  
  5.     <configuration>  
  6.         <webDefaultXml>src/test/resources/webdefault.xml</webDefaultXml>  
  7.         <systemProperties>  
  8.             <systemProperty>  
  9.                 <name>jetty.port</name>  
  10.                 <value>8081</value>  
  11.             </systemProperty>  
  12.         </systemProperties>  
  13.         <contextPath>/</contextPath>  
  14.         <scanIntervalSeconds>3000</scanIntervalSeconds>  
  15.         <scanTargets>  
  16.             <scanTarget>src/main/webapp/WEB-INF</scanTarget>  
  17.             <scanTarget>../target/classes</scanTarget>  
  18.         </scanTargets>  
  19.     </configuration>  
  20. </plugin>
 保存退出,重新编译即可。
最后编译成功:
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-minicluster ---
[INFO] Building jar: /data/release-2.2.0/hadoop-minicluster/target/hadoop-minicluster-2.2.0-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................ SUCCESS [  1.386 s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [  1.350 s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [  2.732 s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [  0.358 s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [  2.048 s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [  3.450 s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [ 16.114 s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [ 13.317 s]
[INFO] Apache Hadoop Common .............................. SUCCESS [05:22 min]
[INFO] Apache Hadoop NFS ................................. SUCCESS [ 16.925 s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [  0.044 s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [02:51 min]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [ 28.601 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [ 27.589 s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [  3.966 s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [  0.044 s]
[INFO] hadoop-yarn ....................................... SUCCESS [ 52.846 s]
[INFO] hadoop-yarn-api ................................... SUCCESS [ 41.700 s]
[INFO] hadoop-yarn-common ................................ SUCCESS [ 25.945 s]
[INFO] hadoop-yarn-server ................................ SUCCESS [  0.105 s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [  8.436 s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [ 15.659 s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [  3.647 s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [ 12.495 s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [  0.684 s]
[INFO] hadoop-yarn-client ................................ SUCCESS [  5.266 s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [  0.102 s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [  2.666 s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [  0.093 s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [ 20.092 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [  2.783 s]
[INFO] hadoop-yarn-site .................................. SUCCESS [  0.225 s]
[INFO] hadoop-yarn-project ............................... SUCCESS [ 36.636 s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [ 16.645 s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [  3.058 s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [  9.441 s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [  5.482 s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [  7.615 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [  2.473 s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [  6.183 s]
[INFO] hadoop-mapreduce .................................. SUCCESS [  6.454 s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [  4.802 s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [ 27.635 s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [  2.850 s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [  6.092 s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [  4.742 s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [  3.155 s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [  3.317 s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [  9.791 s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [  2.680 s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [  0.036 s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [ 20.765 s]
[INFO] Apache Hadoop Client .............................. SUCCESS [  6.476 s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [  0.215 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:32 min
[INFO] Finished at: 2014-07-18T01:18:24+08:00
[INFO] Final Memory: 117M/314M
[INFO] ------------------------------------------------------------------------
 
此时编译好的文件位于 ~/hadoop-dist/target/hadoop-2.2.0/ 目录中
拷贝hadoop-2.2.0到安装目录下,/usr/local/ 重新修改其配置文件,重新并格式化,启动,即可。
到此已经不会报错,可以使用。
[root@db96 hadoop]# hadoop dfs -put ./in
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

put: `.': No such file or directory
[root@db96 hadoop]# file /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 
/usr/local/hadoop/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
 
测试使用:上传一个文件,下载一个文件,查看上传文件的内容:

[root@db96 ~]# cat wwn.txt 
# This is a text txt
# by coco
# 2014-07-18
[root@db96 ~]# hdfs dfs -mkdir /test
[root@db96 ~]# hdfs dfs -put wwn.txt /test
[root@db96 ~]# hdfs dfs -cat /test/wwn.txt
[root@db96 ~]# hdfs dfs -get /test/wwn.txt /tmp
[root@db96 hadoop]# hdfs dfs -rm /test/wwn.txt
[root@db96 tmp]# ll
总用量 6924
-rw-r--r-- 1 root root      70 7月  18 11:50 wwn.txt
[root@db96 ~]# hadoop dfs -ls /test           
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Found 2 items
-rw-r--r--   2 root supergroup    6970105 2014-07-18 11:44 /test/gc_comweight.txt
-rw-r--r--   2 root supergroup         59 2014-07-18 14:56 /test/hello.txt
到此我们的hdfs文件系统已经能正常使用。
 
2015.9.9 20:52 willem 其中还有编译失败的问题需要解决,并记录解决方案。
posted @ 2015-10-30 15:04  离线中  阅读(878)  评论(0编辑  收藏  举报