Hadoop3.1.2 + Hbase2.2.0 设置lzo压缩算法

Hadoop3.1.2 + Hbase2.2.0 设置lzo压缩算法：

写在前面，在配置hbase使用lzo算法时，在网上搜了很多文章，一般都是比较老的文章，一是版本低，二是一般都是使用hadoop-gpl-compression，hadoop-gpl-compression是一个比较老的依赖包，现已被hadoop-lzo替代，希望遇到hadoop和hbase配置lzo算法时，能有所帮助

安装lzo库

1.下载最新的lzo库，下载地址：http://www.oberhumer.com/opensource/lzo/download/

2.解压lzo库

tar -zxvf lzo-2.10.tar.gz

3.进入解压后的lzo目录，执行./configure  --enable-shared

cd lzo-2.10
./configure  --enable-shared -prefix=/usr/local/hadoop/lzo

4.执行make进行编译，编译完成后，执行make install进行安装

make && make install

如果没有安装lzo库，在hbase中创建表时指定compression为lzo时会报错：

ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks
	at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:2314)
	at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2156)
	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:2048)
	at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:651)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103)
	at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2384)
	at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2377)
	at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2154)
	... 7 more
Caused by: java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:355)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98)
	... 10 more

5,库文件被默认安装到了/usr/local/lib，将/usr/local/lib拷贝到/usr/lib下，或者在/usr/lib下建立软连接

cd /usr/lib
ln -s /usr/local/lib/* .

6.安装lzop  wget http://www.lzop.org/download/lzop-1.04.tar.gz

tar -zxvf lzop-1.04.tar.gz 
./configure -enable-shared -prefix=/usr/local/hadoop/lzop
make && make install 

7.把lzop复制到/usr/bin/或建立软连接

ln -s /usr/local/hadoop/lzop/bin/lzop /usr/bin/lzop

 

二、安装hadoop-lzo

1.下载hadoop-lzo ，下载地址：wget https://github.com/twitter/hadoop-lzo/archive/master.zip 这是一个zip压缩包，如果想使用git下载，可以使用该链接：https://github.com/twitter/hadoop-lzo

2.编译hadoop-lzo源码，在编译之前如果没有安装maven需要配置maven环境，解压缩master.zip，为：hadoop-lzo-master，进入hadoop-lzo-master中，修改pom.xml中hadoop版本配置，进行maven编译

unzip master.zip
cd hadoop-lzo-master
vim pom.xml
修改hadoop.current.version为自己对应的hadoop版本，我这里是3.1.2
<properties> 
　　<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> 
　　<hadoop.current.version>3.1.2</hadoop.current.version> 
　　<hadoop.old.version>1.0.4</hadoop.old.version> 
</properties>

3.在hadoop-lzo-master目录中执行一下命令编译hadoop-lzo:

export CFLAGS=-m64
export CXXFLAGS=-m64
export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include   #对应lzo安装的目录
export LIBRARY_PATH=/usr/local/hadoop/lzo/lib　　　　　　#对应lzo安装的目录

mvn clean package -Dmaven.test.skip=true

4.打包完成后，进入target/native/Linux-amd64-64，将libgplcompression*复制到hadoop的native中，将hadoop-lzo.xxx.jar 复制到每台hadoop的common包里

cd target/native/Linux-amd64-64
tar -cBf - -C lib . | tar -xBvf - -C ~
cp ~/libgplcompression* $HADOOP_HOME/lib/native/
cp target/hadoop-lzo-0.4.18-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/

libgplcompression*文件：

 

 

　　其中libgplcompression.so和libgplcompression.so.0是链接文件，指向libgplcompression.so.0.0.0，

将上面生成的libgplcompression*和target/hadoop-lzo-xxx-SNAPSHOT.jar同步到集群中的所有机器对应的目录($HADOOP_HOME/lib/native/，$HADOOP_HOME/share/hadoop/common/)。

配置hadoop环境变量

1.在$HADOOP_HOME/etc/hadoop/hadoop-env.sh文件中配置：

export LD_LIBRARY_PATH=/usr/local/lib/lzo/lib
# Extra Java CLASSPATH elements.  Optional.
export HADOOP_CLASSPATH="<extra_entries>:$HADOOP_CLASSPATH:${HADOOP_HOME}/share/hadoop/common"
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_HOME/lib/native

2.在$HADOOP_HOME/etc/hadoop/core-site.xml加上如下配置：

<property>
    <name>io.compression.codecs</name>
    <value>org.apache.hadoop.io.compress.GzipCodec,
           org.apache.hadoop.io.compress.DefaultCodec,
           com.hadoop.compression.lzo.LzoCodec,
           com.hadoop.compression.lzo.LzopCodec,
           org.apache.hadoop.io.compress.BZip2Codec
        </value>
</property>

<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

如果没有配置，在hbase中创建表时compression指定lzo时会报错：

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103)
	at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2384)
	at org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:2377)
	at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:2154)
	... 7 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:128)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:114)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:353)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98)
	... 10 more
Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:124)
	... 13 more

3.在$HADOOP_HOME/etc/hadoop/mapred-site.xml加上如下配置：

<property>
    <name>mapred.compress.map.output</name>
    <value>true</value>
</property>

<property>
    <name>mapred.map.output.compression.codec</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

<property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib</value>
</property>

<property>
    <name>mapreduce.reduce.env</name>
    <value>LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib</value>
</property>

<property>
    <name>mapred.child.env</name>
    <value>LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib</value>
</property>

将上述修改的配置文件全部同步到集群的所有机器上，并重启Hadoop集群，这样就可以在Hadoop中使用lzo。

在Hbase中配置lzo

 

1.将hadoop-lzo-xxx.jar复制到/hbase/lib中

cp target/hadoop-lzo-0.4.18-SNAPSHOT.jar $HBASE_HOME/lib

2.在hbase/lib下创建native文件夹，在/hbase/lib/native下创建Linux-amd64-64 -> /opt/hadoop/lib/native的软连接

ln -s /opt/hadoop/lib/native Linux-amd64-64

如图：

 

 

3.在$HBASE_HOME/conf/hbase-env.sh中添加如下配置：

export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/

4.在$HBASE_HOME/conf/hbase-site.xml中添加如下配置：

<property>
     <name>hbase.regionserver.codecs</name>
     <value>lzo</value>
</property>

5.启动hbase一切正常

 

注意：关于hadoop-gpl-compression的说明：

　　hadoop-lzo-xxx的前身是hadoop-gpl-compression-xxx,之前是放在google code下管理,地址:http://code.google.com/p/hadoop-gpl-compression/ .但由于协议问题后来移植到github上,也就是现在的hadoop-lzo-xxx,github,链接地址:https://github.com/kevinweil/hadoop-lzo.网上介绍hadoop lzo压缩绝大部分都是基于hadoop-gpl-compression的介绍.而hadoop-gpl-compression还是09年开发的,跟现在hadoop版本已经无法再完全兼容,会发生一些问题。因此也趟了一些坑。希望能给一些朋友一点帮助。

在使用hadoop-gpl-compression-xxx.jar时，hbase启动会报如下错：

2019-09-03 11:36:22,771 INFO  [main] lzo.GPLNativeCodeLoader: Loaded native gpl library
2019-09-03 11:36:22,866 WARN  [main] lzo.LzoCompressor: java.lang.NoSuchFieldError: lzoCompressLevelFunc
2019-09-03 11:36:22,866 ERROR [main] lzo.LzoCodec: Failed to load/initialize native-lzo library
2019-09-03 11:36:23,169 WARN  [main] util.CompressionTest: Can't instantiate codec: lzo
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:103)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:69)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:834)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:565)
	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:506)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)
Caused by: java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:355)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:98)
	... 14 more
2019-09-03 11:36:23,183 ERROR [main] regionserver.HRegionServer: Failed construction RegionServer
java.io.IOException: Compression codec lzo not supported, aborting RS construction
	at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:835)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:565)
	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:506)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)
2019-09-03 11:36:23,184 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster. 
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3187)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)
Caused by: java.io.IOException: Compression codec lzo not supported, aborting RS construction
	at org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs(HRegionServer.java:835)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:565)
	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:506)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:3180)
	... 5 more

　　当删除hadoop-gpl-compression-xxx.jar时，替换为hadoop-lzo.xxx.jar后，再启动hbase，一切正常：

2019-09-03 14:57:43,755 INFO  [main] lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries
2019-09-03 14:57:43,758 INFO  [main] lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 5dbdddb8cfb544e58b4e0b9664b9d1b66657faf5]
2019-09-03 14:57:43,983 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2019-09-03 14:57:44,088 INFO  [main] compress.CodecPool: Got brand-new compressor [.lzo_deflate]