Perl 通过Thrift连接Hbase

本文首发http://f.dataguru.cn/thread-217843-1-1.html。

 

一、下载安装thrift

下载thrift-0.9.1.tar.gz安装包

使用tar -xvf thrift-0.9.1.tar.gz解压该包

cd thrift-0.9.1

按照官方文档上的instal thrift 下载相应依赖包，地址为:

http://thrift.apache.org/docs/install/ubuntu/

     执行如下命令 ：

sudo apt-get install libboost-dev libboost-test-dev libboost-program-options-dev           libevent-dev automake libtool flex bison pkg-config g++ libssl-dev

 

增加perl的依赖包:

sudo apt-get install libbit-vector-perl

 

在~/.bashrc中添加新的环境变量(为以后好编译Perl)

export THRIFT_HOME=/home/grid/thrift-0.9.1

 

对thrift进行安装

./configure

 

grid@h01:~/thrift-0.9.1$ ./configure

checking for a BSD-compatible install... /usr/bin/install -c

checking whether build environment is sane... yes

checking for a thread-safe mkdir -p... /bin/mkdir -p

checking for gawk... no

checking for mawk... mawk

checking whether make sets $(MAKE)... yes

checking how to create a ustar tar archive... gnutar

checking for pkg-config... /usr/bin/pkg-config

checking pkg-config is at least version 0.9.0... yes

checking for gcc... Gcc

....

thrift 0.9.1

 

Building C++ Library ......... : yes

Building C (GLib) Library .... : yes

Building Java Library ........ : no

Building C# Library .......... : no

Building Python Library ...... : yes

Building Ruby Library ........ : no

Building Haskell Library ..... : no

Building Perl Library ........ : yes

Building PHP Library ......... : no

Building Erlang Library ...... : no

Building Go Library .......... : no

Building D Library ........... : no

 

C++ Library:

   Build TZlibTransport ...... : yes

   Build TNonblockingServer .. : yes

   Build TQTcpServer (Qt) .... : no

 

Python Library:

   Using Python .............. : /usr/bin/python

 

Perl Library:

   Using Perl ................ : /usr/bin/perl

 

If something is missing that you think should be present,

please skim the output of configure to find the missing

component.  Details are present in config.log.

grid@h01:~/thrift-0.9.1$ make&make install

[1] 41403

make  all-recursive

Making install in compiler/cpp

make[1]: Entering directory `/home/grid/thrift-0.9.1'

Making all in compiler/cpp

make[2]: Entering directory `/home/grid/thrift-0.9.1/compiler/cpp'

make[1]: Entering directory `/home/grid/thrift-0.9.1/compiler/cpp'

make  install-am

make  all-am

make[3]: Entering directory `/home/grid/thrift-0.9.1/compiler/cpp'

make[2]: Entering directory `/home/grid/thrift-0.9.1/compiler/cpp'

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-main.o -MD -MP -MF .deps/thrift-main.Tpo -c -o thrift-main.o `test -f 'src/main.cc' || echo './'`src/main.cc

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-main.o -MD -MP -MF .deps/thrift-main.Tpo -c -o thrift-main.o `test -f 'src/main.cc' || echo './'`src/main.cc

mv -f .deps/thrift-main.Tpo .deps/thrift-main.Po

mv -f .deps/thrift-main.Tpo .deps/thrift-main.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_c_glib_generator.o -MD -MP -MF .deps/thrift-t_c_glib_generator.Tpo -c -o thrift-t_c_glib_generator.o `test -f 'src/generate/t_c_glib_generator.cc' || echo './'`src/generate/t_c_glib_generator.cc

mv: cannot stat `.deps/thrift-main.Tpo': No such file or directory

make[3]: *** [thrift-main.o] Error 1

make[3]: Leaving directory `/home/grid/thrift-0.9.1/compiler/cpp'

make[2]: *** [all] Error 2

make[2]: Leaving directory `/home/grid/thrift-0.9.1/compiler/cpp'

make[1]: *** [all-recursive] Error 1

make[1]: Leaving directory `/home/grid/thrift-0.9.1'

make: *** [all] Error 2

mv -f .deps/thrift-t_c_glib_generator.Tpo .deps/thrift-t_c_glib_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_cpp_generator.o -MD -MP -MF .deps/thrift-t_cpp_generator.Tpo -c -o thrift-t_cpp_generator.o `test -f 'src/generate/t_cpp_generator.cc' || echo './'`src/generate/t_cpp_generator.cc

mv -f .deps/thrift-t_cpp_generator.Tpo .deps/thrift-t_cpp_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_java_generator.o -MD -MP -MF .deps/thrift-t_java_generator.Tpo -c -o thrift-t_java_generator.o `test -f 'src/generate/t_java_generator.cc' || echo './'`src/generate/t_java_generator.cc

mv -f .deps/thrift-t_java_generator.Tpo .deps/thrift-t_java_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_as3_generator.o -MD -MP -MF .deps/thrift-t_as3_generator.Tpo -c -o thrift-t_as3_generator.o `test -f 'src/generate/t_as3_generator.cc' || echo './'`src/generate/t_as3_generator.cc

mv -f .deps/thrift-t_as3_generator.Tpo .deps/thrift-t_as3_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_csharp_generator.o -MD -MP -MF .deps/thrift-t_csharp_generator.Tpo -c -o thrift-t_csharp_generator.o `test -f 'src/generate/t_csharp_generator.cc' || echo './'`src/generate/t_csharp_generator.cc

mv -f .deps/thrift-t_csharp_generator.Tpo .deps/thrift-t_csharp_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_py_generator.o -MD -MP -MF .deps/thrift-t_py_generator.Tpo -c -o thrift-t_py_generator.o `test -f 'src/generate/t_py_generator.cc' || echo './'`src/generate/t_py_generator.cc

mv -f .deps/thrift-t_py_generator.Tpo .deps/thrift-t_py_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_rb_generator.o -MD -MP -MF .deps/thrift-t_rb_generator.Tpo -c -o thrift-t_rb_generator.o `test -f 'src/generate/t_rb_generator.cc' || echo './'`src/generate/t_rb_generator.cc

mv -f .deps/thrift-t_rb_generator.Tpo .deps/thrift-t_rb_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_perl_generator.o -MD -MP -MF .deps/thrift-t_perl_generator.Tpo -c -o thrift-t_perl_generator.o `test -f 'src/generate/t_perl_generator.cc' || echo './'`src/generate/t_perl_generator.cc

mv -f .deps/thrift-t_perl_generator.Tpo .deps/thrift-t_perl_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_php_generator.o -MD -MP -MF .deps/thrift-t_php_generator.Tpo -c -o thrift-t_php_generator.o `test -f 'src/generate/t_php_generator.cc' || echo './'`src/generate/t_php_generator.cc

mv -f .deps/thrift-t_php_generator.Tpo .deps/thrift-t_php_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_erl_generator.o -MD -MP -MF .deps/thrift-t_erl_generator.Tpo -c -o thrift-t_erl_generator.o `test -f 'src/generate/t_erl_generator.cc' || echo './'`src/generate/t_erl_generator.cc

mv -f .deps/thrift-t_erl_generator.Tpo .deps/thrift-t_erl_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_cocoa_generator.o -MD -MP -MF .deps/thrift-t_cocoa_generator.Tpo -c -o thrift-t_cocoa_generator.o `test -f 'src/generate/t_cocoa_generator.cc' || echo './'`src/generate/t_cocoa_generator.cc

mv -f .deps/thrift-t_cocoa_generator.Tpo .deps/thrift-t_cocoa_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_st_generator.o -MD -MP -MF .deps/thrift-t_st_generator.Tpo -c -o thrift-t_st_generator.o `test -f 'src/generate/t_st_generator.cc' || echo './'`src/generate/t_st_generator.cc

mv -f .deps/thrift-t_st_generator.Tpo .deps/thrift-t_st_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_ocaml_generator.o -MD -MP -MF .deps/thrift-t_ocaml_generator.Tpo -c -o thrift-t_ocaml_generator.o `test -f 'src/generate/t_ocaml_generator.cc' || echo './'`src/generate/t_ocaml_generator.cc

mv -f .deps/thrift-t_ocaml_generator.Tpo .deps/thrift-t_ocaml_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_hs_generator.o -MD -MP -MF .deps/thrift-t_hs_generator.Tpo -c -o thrift-t_hs_generator.o `test -f 'src/generate/t_hs_generator.cc' || echo './'`src/generate/t_hs_generator.cc

mv -f .deps/thrift-t_hs_generator.Tpo .deps/thrift-t_hs_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_js_generator.o -MD -MP -MF .deps/thrift-t_js_generator.Tpo -c -o thrift-t_js_generator.o `test -f 'src/generate/t_js_generator.cc' || echo './'`src/generate/t_js_generator.cc

mv -f .deps/thrift-t_js_generator.Tpo .deps/thrift-t_js_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_javame_generator.o -MD -MP -MF .deps/thrift-t_javame_generator.Tpo -c -o thrift-t_javame_generator.o `test -f 'src/generate/t_javame_generator.cc' || echo './'`src/generate/t_javame_generator.cc

mv -f .deps/thrift-t_javame_generator.Tpo .deps/thrift-t_javame_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_delphi_generator.o -MD -MP -MF .deps/thrift-t_delphi_generator.Tpo -c -o thrift-t_delphi_generator.o `test -f 'src/generate/t_delphi_generator.cc' || echo './'`src/generate/t_delphi_generator.cc

mv -f .deps/thrift-t_delphi_generator.Tpo .deps/thrift-t_delphi_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_go_generator.o -MD -MP -MF .deps/thrift-t_go_generator.Tpo -c -o thrift-t_go_generator.o `test -f 'src/generate/t_go_generator.cc' || echo './'`src/generate/t_go_generator.cc

mv -f .deps/thrift-t_go_generator.Tpo .deps/thrift-t_go_generator.Po

g++ -DHAVE_CONFIG_H -I. -I../.. -I../../lib/cpp/src/thrift  -I./src  -Wall -g -O2 -MT thrift-t_d_generator.o -MD -MP -MF .deps/thrift-t_d_generator.Tpo -c -o thrift-t_d_generator.o `test -f 'src/generate/t_d_generator.cc' || echo './'`src/generate/t_d_generator.cc

mv -f .deps/thrift-t_d_generator.Tpo .deps/thrift-t_d_generator.Po

/bin/bash ../../libtool --tag=CXX   --mode=link g++ -Wall -g -O2   -o thrift thrift-main.o thrift-md5.o thrift-t_generator.o thrift-parse.o thrift-t_c_glib_generator.o thrift-t_cpp_generator.o thrift-t_java_generator.o thrift-t_as3_generator.o thrift-t_csharp_generator.o thrift-t_py_generator.o thrift-t_rb_generator.o thrift-t_perl_generator.o thrift-t_php_generator.o thrift-t_erl_generator.o thrift-t_cocoa_generator.o thrift-t_st_generator.o thrift-t_ocaml_generator.o thrift-t_hs_generator.o thrift-t_xsd_generator.o thrift-t_html_generator.o thrift-t_js_generator.o thrift-t_javame_generator.o thrift-t_delphi_generator.o thrift-t_go_generator.o thrift-t_gv_generator.o thrift-t_d_generator.o -lfl libparse.a

libtool: link: g++ -Wall -g -O2 -o thrift thrift-main.o thrift-md5.o thrift-t_generator.o thrift-parse.o thrift-t_c_glib_generator.o thrift-t_cpp_generator.o thrift-t_java_generator.o thrift-t_as3_generator.o thrift-t_csharp_generator.o thrift-t_py_generator.o thrift-t_rb_generator.o thrift-t_perl_generator.o thrift-t_php_generator.o thrift-t_erl_generator.o thrift-t_cocoa_generator.o thrift-t_st_generator.o thrift-t_ocaml_generator.o thrift-t_hs_generator.o thrift-t_xsd_generator.o thrift-t_html_generator.o thrift-t_js_generator.o thrift-t_javame_generator.o thrift-t_delphi_generator.o thrift-t_go_generator.o thrift-t_gv_generator.o thrift-t_d_generator.o  -lfl libparse.a

make[3]: Entering directory `/home/grid/thrift-0.9.1/compiler/cpp'

test -z "/usr/local/bin" || /bin/mkdir -p "/usr/local/bin"

  /bin/bash ../../libtool   --mode=install /usr/bin/install -c thrift '/usr/local/bin'

libtool: install: /usr/bin/install -c thrift /usr/local/bin/thrift

make[3]: Nothing to be done for `install-data-am'.

make[3]: Leaving directory `/home/grid/thrift-0.9.1/compiler/cpp'

make[2]: Leaving directory `/home/grid/thrift-0.9.1/compiler/cpp'

make[1]: Leaving directory `/home/grid/thrift-0.9.1/compiler/cpp'

Making install in lib

make[1]: Entering directory `/home/grid/thrift-0.9.1/lib'

Making install in cpp

make[2]: Entering directory `/home/grid/thrift-0.9.1/lib/cpp'

Making install in .

make[3]: Entering directory `/home/grid/thrift-0.9.1/lib/cpp'

make[4]: Entering directory `/home/grid/thrift-0.9.1/lib/cpp'

test -z "/usr/local/lib" || /bin/mkdir -p "/usr/local/lib"

/bin/bash ../../libtool   --mode=install /usr/bin/install -c   libthrift.la libthriftnb.la libthriftz.la '/usr/local/lib'

libtool: install: /usr/bin/install -c .libs/libthrift-0.9.1.so /usr/local/lib/libthrift-0.9.1.so

/usr/bin/install: cannot remove `/usr/local/lib/libthrift-0.9.1.so': Permission denied

make[4]: *** [install-libLTLIBRARIES] Error 1

make[4]: Leaving directory `/home/grid/thrift-0.9.1/lib/cpp'

make[3]: *** [install-am] Error 2

make[3]: Leaving directory `/home/grid/thrift-0.9.1/lib/cpp'

make[2]: *** [install-recursive] Error 1

make[2]: Leaving directory `/home/grid/thrift-0.9.1/lib/cpp'

make[1]: *** [install-recursive] Error 1

make[1]: Leaving directory `/home/grid/thrift-0.9.1/lib'

make: *** [install-recursive] Error 1

[1]+  Exit 2                  make

grid@h01:~$ thrift -version

Thrift version 0.9.1

 

 

这个过程中出现了一些目录没有找到和权限不足的情况，有同学说是没有执行安装目录，尝试过在.configure 后加./configure --prefix=/home/grid/thrift也没有生效，欢迎朋友指教怎么设置参数规避这个问题。

 

二、用Perl进行连接Hbase

生成先关文件:

thrift --gen perl  \

/home/grid/hbase-0.94.13/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift

可以看到生成了相应的Perl包

grid@h01:~/thrift-0.9.1/gen-perl/Hbase$ ls

Constants.pm  Hbase.pm  Types.pm

接着设置环境变量（这里表示Hbase Perl环境变量的意思,哈哈)

Export HP_HOME=~/thrift-0.9.1/gen-perl/

 

启动hadoop和hbase/

grid@h01:~/wangn$ jps

7622 HQuorumPeer

3612 DataNode

4196 TaskTracker

3341 NameNode

35516 HMaster

35785 HRegionServer

3957 JobTracker

42656 ThriftServer

49545 Jps

3856 SecondaryNameNode

 

基于这个目录里的Demo来完成实验,这里面的例子相对来说应该是最全的，是很好的学习材料。11

grid@h01:~/hbase-0.94.13/src/examples/thrift$ ls

DemoClient.cpp  DemoClient.java  DemoClient.php  DemoClient.pl  DemoClient.py  DemoClient.rb  Makefile  README.txt

Cp DemoClient.pl hclient.pl

 

在hclient.pl中添加红色部分。

use strict;

use warnings;

 

use Env qw(HP_HOME);

use Env qw(THRIFT_HOME);

 

use lib "$HP_HOME/lib";

use lib "$THRIFT_HOME/lib/perl/lib";

 

use Thrift::Socket;

use Thrift::BufferedTransport;

use Thrift::BinaryProtocol;

use Hbase::Hbase;

use Data:umper;

 

执行perl hclient.pl 此时有可能会得到一个:

Base class package "Class::Accessor" is empty.的异常，说明缺少依赖的包。

下载 ：http://search.cpan.org/~kasei/Class-Accessor-0.34/lib/Class/Accessor.pm 将其中的lib中的Class目录放        在@INC的目录中，此时编译应该没有问题了。

 

此时有可能会得到如下错误:

column families in {demo_table}:

  column: {entry:}, maxVer: {10}

  column: {unused:}, maxVer: {3}

Can't use string ("shouldn't get here! at hclient.p"...) as a HASH ref while "strict refs" in use at hclient.pl line 166.

表示作者的的一个假设没有成立，可能和hbase的版本有关。

我们注释162行:#die ("shouldn't get here!";

 

从新执行 perl hclient.pl

perl hclient.pl

scanning tables...

  found {demo_table}

    disabling table: {demo_table}

    deleting table: {demo_table}

  found {member}

  found {person}

  found {test01}

creating table: {demo_table}

 

。。。。

会列出我们hbase里的表和创建新表插入数据，扫描数据等操作。