hadoop安装snappy

Hadoop安装Snappy
安装编译依赖部分：
第一步：安装Maven
将MAVEN_HOME配置到环境变量里面
需要再settings.xml配置文件中增加以下内容
<profile>
<id>jdk-1.7</id>
<activation>
<jdk>1.7</jdk>
</activation>
<repositories>
<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.oschina.net/content/groups/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</profile>
第二步：安装gcc
yum -y install gcc*
第三步：安装ncurses-devel，openssl-devel，cmake
yum -y install ncurses-devel openssl-devel cmake
第四步：安装protobuf
解压缩：tar -zxvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0/
./configure
make
make check
make install
第五步：下载snappy源码
wget ftp://ftp.netbsd.org/pub/pkgsrc/distfiles/snappy-1.1.1.tar.gz
第六步：安装相关依赖
yum install -y automake autoconf libtool
第七步：编译snappy源码
tar -zxvf snappy-1.1.1.tar.gz
cd snappy-1.1.1
./configure
make
make check
make install
编译完成之后会将snappy相关.so链接库默认安装到/usr/local/lib目录下面
编译hadoop
在编译的时候，需要指定使用snappy
mvn package -Pdist,native -DskipTests -Dtar -Drequire.snappy
#mvn clean package -Pdist,native -DskipTests -Dtar -Drequire.snappy -Dbundle.snappy -Dsnappy.lib=/usr/local/lib
编译好之后，将原来的$HADOOP_HOME/lib/native替换为编译好的新的native库
然后拷贝编译后的snappy.so lib库到$HADOOP_HOME/lib/native/中
编译hadoop-snappy
mvn package -DskipTests
将编译好的hadoop-snappy-0.0.1-SNAPSHOT.jar拷贝到$HADOOP_HOME/lib
修改hadoop配置文件
1°、修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh，添加：
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/Linux-amd64-64/
2°、修改$HADOOP_HOME/etc/hadoop/core-site.xml：
<property>
<name>io.compression.codecs</name>
<value>
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
org.apache.hadoop.io.compress.SnappyCodec
</value>
</property>
3°、修改$HADOOP_HOME/etc/hadoop/mapred-site.xml中有关压缩属性，测试snappy：
<property>
<name>mapreduce.map.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.output.compress.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
4°、验证
hadoop checknative -a