在CentOS上安装Thrift
在CentOS上安装Thrift
http://wiki.apache.org/thrift/ThriftRequirements
http://wiki.apache.org/thrift/ThriftInstallation
Step1：安装依赖
sudo yum install automake libtool flex bison pkgconfig gcc-c++ boost-devel libevent-devel zlib-devel python-devel ruby-devel
Step2：下载和编译Thrift
wget http://www.eu.apache.org/dist//thrift/0.7.0/thrift-0.7.0.tar.gz
tar -zxvf thrift-0.7.0.tar.gz cd thrift-0.7.0 ./configure; make; make install
Step3：生成运行时（Runtime）
cd ./lib/python python setup.py install #生成python运行时
或
cd ./lib/java ... #生成libthrift.jar,thrift的java运行时
或
cd ./lib/php ... ##生成php运行时
Thrift 学习小记 http://www.cnblogs.com/yuhui962/archive/2011/08/08/2131206.html
1. 基本名词
1. Types： 为了满足多语言平台的要求，需要提供基本数据类型来进行转换。比如在C++的Map和Python的Dict之间能够相互转换。
2. Transport: 对于每一种语言，都应该有一个抽象的公共层来完成对应的双向数据传输。
3. Protocal： 数据需要有一种方式来使用对应的传输层的code，而不用关心传输层的具体实现细节。
4. Versioning：数据需要有自己的版本号来实现对应的健壮性。
5. Processing ： 产生code来完成RPC调用。
2. 类型
1. Goals：
1). 使用最基本的数据类型，不管上层使用怎么样的编程语言。
2). 不使用动态数据类型，也不需要开发者写对象序列化或者传输的代码(已经封装)。
3). IDL 用来告诉代码生成器如何在多语言之上安全地传输数据。
2. Base Types：
1). 选取了大多数语言中都存在的基本数据类型，忽略了一些语言中特别的类型。(无符号类型)， 所有的基本类型都是有符号的。
2). 基本类型包括bool， byte，i16，i32，i64，double，string。
3. Structs：
1). 提供了一种common的对象来代表面向对象语言中的对象类型，每一个filed都有一个标志符。
2). 每个filed必须有一个标志符和一个给定的默认值。标志符如果没有给定的话，会自动分配一个，但是强烈建议给定标志符，为了方便版本控制。
4. Container：
1). List， set， Map。其中List被翻译成STL中的vector， 或者Java中的ArrayList，set和map类似。
2). 要求对应的基本语言类型支持对应的Thrift的基本数据类型。
3). 在每一种目标语言中，对于每一种定义的类型，生成两个接口，read和write，用于序列化和传输对应的数据结构。
5. Exceptions：
1). 使用Exception关键字来代替对应的structs关键字，得到对应的基本类。
2). 生成的对象继承自每一种语言中的对应的Exception的基类。
6. Services：
1). 提供了服务端的接口调用，通过代码生成器会生成Client端和Server端的桩函数接口，共上层调用使用。
2). 在每一个函数前面可以加上async关键字，这个关键字能够实现对应的一部方法。
3). 一个纯void函数也会返回一个response，用来保证该操作在服务端已经被执行。
4). async call只保证消息在传输层是没有问题的，因此，在使用async的时候需要保证当前的使用环境对于丢包是可以接受的。
3. Transport
1. 接口
1). 传输层用来促进数据传输。一个关键的设计是Thrift将数据传输层和代码生成器分开。
2). 由于抽象的IO炒作产生的tradoff相比起实际的IO操作显得微不足道。
3). 需要满足的基本功能：如何读写数据，不管是在网络中，还是共享内存，还是从磁盘文件中读写。
4). 提供的接口如下：open， close，isOpen, read, write, flush。还有一些用于batch操作的接口
5). 对于TServerTransport，还有接口用来创建对应的Transport对象。如open， listen, accept, close。
2. 实现
1). TSocket 类对于TCP/IP的流式套接字实现了一个通用，简单的接口。
2). TFileTransport类实现了一个抽象接口，用于将磁盘文件导入到数据流中。
3. Utilities
1). 网络传输层设计的支持简单的扩展，比如组合，TBufferdTransport实现在传输层实现了buffer的功能，TFrameTransport 实现了数据头定长大小，用于Noblocking传输。TMemoryBuffer允许读和写直接在进程的堆顶完成。
4. Protocal
1. 接口
1). 另外一块主要的部分是thrift中将数据结构和传输层分离，thrift定义了一些传输时的基本类型，但是没有强制定义需要实现的编码格式，只要能被代码生成器识别就好。
2). thrift协议的接口简单明了，支持两种功能：双向有序的消息，基本类型，容器，结构的编码。
2. 结构
1). Thrift中默认使用流式结构来传输对应的协议，这样避免了因为需要分片或者在发送前整体计算带来的性能损失。当有的场景需要使用分片，或者分片的性能优势更加明显的时候，可以使用TFramedTransport这个抽象接口类。
3. 实现
1). Thrift的实现了一个空间高效的二进制协议，所有的数据都以一个扁平的二进制格式存在，整形被转换为网络字节序，字符串在头上加上它的长度，所有的消息和field 头都用它的整数序列号，忽略这些filed中的字符串名字。
2). 我们没有做过多的性能优化，为了代码的整洁和高效。如果有需要，可以加入这些优化方案。
5.Versioning
1. 域标志符
1). Thrift在版本控制上面是健壮的，它可以接受旧客户端过来的请求并正确处理。
2). Thrift中的版本控制是通过域标志符来确定的，thrift结构中的每一项前面都有一个对应的标志符，标志符和数据类型唯一化开item。
3). 为了避免冲突，自动分配的标志符从-1开始递减，人工定义的标志符只能为整数。
4). 在数据是自定义的前提下，当反序列化的时候，thrift的code 生成器可以根据标识符来判断读取到的数据是否对齐，也可以跳过不能识别的域，保证兼容性。
5). 域标志符也可以在参数列表中声。
2. Isset
1). 用来标注必须存在于结构体中的域，如果必须存在，标注为true，这样可以当该域不存在的时候，可以通知上层调用者。
3. 场景分析
1). 4种场景，加减域/新旧client或者server。
4. Protocal/Transport Versioning
1). 实现了TProtocal的抽象接口，用来让协议的实现自己来管理版本。
6. RPC 实现
1. TProcessor
1). 核心类，抽象出client和server的基类，核心功能是处理input和output。
2. Generated code
2). 自动生成client端和server端的调用接口。
3. TServer
1). Tserver 可以有多种实现，来满足不同的需求。如单线程的TSimpleServer, 每个连接一个线程的TThreadedServer， 还有基于线程库的TThreadPoolServer.
2). 开发者可以实现自己的TServer。
7. 实现细节
1. 目标语言
1). C++， Java， Python， Ruby， PHP
2). 尽可能简单的抽象，保留在每一种语言中可以自主开发的空间。
2. 生成的结构体
3. RPC 方法定义
1). RPC call中的方法用string去实现，用一些手段将字符处理的代价尽可能的减少，同时方便编程实现。
4. 服务器和多线程
1). 因为种种原因，没有采用boost的线程库，也没有采用ACE，自己实现了一套机制。
5. 线程模型
1). thfirt的线程库包括三个部分：primitives， thread pool manager， timer manager
2). 每个线程是一个对象，提供了Runnerable接口供使用，Synchronized关键字实现类似java的同步块功能。
6. Thread,Runnable, and shared_ptr
7. ThreadManager
8. TimeManager
9. NonBlocking Operation
10. 编译器
11. TFileTransport
8. Facebook thrift service
1. Search&&logging
1). TFileTransport可以实现对应的结构化logging， 这个logging日志可以用来做线上和线下的处理，可以作为redo log。
Thrift的"Hello World!" http://www.thrift.pl/Thrift-tutorial-our-own-hello-world.html
This tutorial "first application" is something, that you can as well deduce from the official Apache Thrift tutorial, although in my opinion, it explains things more clearly.
Our first application using Apache Thrift will be quite simple. We will define object "User". This object will be created in PHP, and stored in the "database" (this means a list:) in Python. One could also retrieve the object from Python and pass it to PHP or clear the list. Although this example is very primitive, it will give you some idea about how the Apache Thrift works.
First, we will define methods and objects, that are going to be exchanged between both applications. These definitions we will put in the file named "hello.thrift". This file has C-like syntax, with some Thrift-specific modifications – as will be explained later.
Thrift files can include other files. In our example we will not include any other files, however in the tutorial project you may notice, that file "shared.thrift" is needed. This file declares the class, that the main class Calculator inherits from. In our example we don't need that.
At the beginning you may define namespaces:
namespace php hello
Namespaces are defined per language – in this example, only objects in PHP file will be prefixed with "hello_".
Object "User" will have fields "firstname" and "lastname" of type string, "user_id" of type integer, "active" of type boolean, and "sex", being our own enum type.
Defining such object is simple. First, we have to define our enum type:
enum SexType {
MALE = 1,
FEMALE = 2
}
And then user object:
struct User {
1: string firstname,
2: string lastname,
3: i32 user_id = 0,
4: SexType sex,
5: bool active = false,
6: optional string description
}
As you may have noticed, we have to number the parameters – that's the convention required by Thrift. By default, all parameters are required, but you can make them optional (by using "optional" keyword), as well as setting their default value.
We may also declare an exception. It will be used, when wrong parameter is passed (in our simple example, it may be, i.e. negative user_id number).
exception InvalidValueException {
1: i32 error_code,
2: string error_msg
}
Now it's time to declare service. Services are the main thing, that makes serialized objects to be available through different programs. Declaration of our simple service is as follows:
service UserExchange {
void ping(),
i32 add_user(1:User u) throws (1: InvalidValue e),
User get_user(1:i32 uid) throws (1: InvalidValue e),
oneway void clear_list()
}
When declaring methods, the same rule of numbering parameters applies. As you see, method declaration is very C-like.
Quick explanation of our methods:
ping – simple ping-pong method. We send "are you there?" message and receive "yes, I am here" response – of course if everything goes well.
add_user – we send user object (notice, that it's an object, so it will be created in PHP application). In response we get this user's id number. If something goes wrong – we have to catch error thrown by this method.
get_user – same as above, but here we supply user_id and receive User object.
clear_list – this function is declared as "oneway" - that means, request will be sent, but our PHP application is not going to wait for the result (as there will be none). Obviously, return value of such function should be void.
Now it's time to generate PHP and Python files, to be used in our client and server, respectively. It is easily done by:
thrift -r --gen php hello.thrift thrift -r --gen py hello.thrift
In the directories "gen-php" and "gen-py" we have PHP and Python files, respectively.
Let's create the server now. We create file, let's say, "python_server.py". Don't forget to make it runnable, by using command:
chmod +x ./python_server.py
In this file we will include all needed libraries, define class to handle the requests, with method names the same as these defined in the thrift file. As the code is self explanatory, I will paste it here:
#!/usr/bin/env python
import sys
sys.path.append('./gen-py')
from hello import UserManager
from hello.ttypes import *
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer
users = []
class UserManagerHandler:
def __init__(self):
pass
#self.log = {}
def ping(self):
print 'ping()'
def add_user(self, user):
if user.firstname == None:
raise InvalidValueException(1,'no firstname exception')
if user.lastname == None:
raise InvalidValueException(2, 'no lastname exception')
if user.user_id <= 0:
raise InvalidValueException(3, 'wrong user_id')
if user.sex != SexType.MALE and user.sex != SexType.FEMALE:
raise InvalidValueException(4, 'wrong sex id')
print 'Processing user '+user.firstname+' '+user.lastname
users.append(user)
print users
return True
def get_user(self, user_id):
if user_id < 0:
raise InvalidValueException(5, 'wrong id')
return users[user_id]
def clear_list(self):
print 'Clearing list'
print users
del users [:]
print users
handler = UserManagerHandler()
processor = UserManager.Processor(handler)
transport = TSocket.TServerSocket(9090)
tfactory = TTransport.TBufferedTransportFactory()
pfactory = TBinaryProtocol.TBinaryProtocolFactory()
server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)
# You could do one of these for a multithreaded server
#server = TServer.TThreadedServer(processor, transport, tfactory, pfactory)
#server = TServer.TThreadPoolServer(processor, transport, tfactory, pfactory)
print 'Starting the server...'
server.serve()
print 'done.'
As you see, this script stores users in the 'users' list. Provided methods allow manipulation of the contents of the list.
Line, that you should take into consideration is:
sys.path.append('./gen-py')
It is path to the "gen-py" directory, relative to the path from where the script would be run (so, in fact, it's best to put here absolute path).
You can run your script by typing:
./python_server.py
You may encounter following error:
ImportError: No module named Thrift
To fix this, you should go to the directory "lib/py" of thrift package and run command to install thrift module in to your python libraries:
sudo python setup.py install
Thanks for the tip on that to Kris Van den Bergh :)
Before creating client PHP script, prepare the structure of the files. First of all, choose directory that is accessible by your Apache (or any other) http server. Then, create "src" directory and put there all the files and directories from the "lib/php/src" directory in the thrift package. Create also "src/packages" directory and put there "hello" directory from the "gen-php" directory (the one generated by thrift). In the main directory put your PHP file (i.e. "hello.php"). In this script you will include Thrift libraries, include your auto-generated UserManager class file, establish connection with the server and perform exchange of the objects. As the contents of the file is self-explanatory to anyone with basic PHP knowledge, I paste it entirely here:
open();
$client->ping();
$u = new hello_User();
$u->user_id = 1;
$u->firstname = 'John';
$u->lastname = 'Smith';
$u->sex = hello_SexType::MALE;
if ($client->add_user($u))
{
echo 'user added succesfully
';
}
var_dump($client->get_user(0));
$client->clear_list();
$u2 = new hello_User();
$client->add_user($u2);
} catch (hello_InvalidValueException $e) {
echo $e->error_msg.'
';
}
?>
That's it! Now run your server (if it's not running already) and open yours PHP file's address in the web browser. You should get result like this:
user added succesfully object(hello_User)[7] public 'firstname' => string 'John' (length=3) public 'lastname' => string 'Smith' (length=8) public 'user_id' => int 1 public 'active' => boolean true public 'sex' => int 1 public 'description' => null no firstname exception
(var_dump's result may be different, if you don't have xdebug installed)
As you analyze contents of the PHP script, you will notice, that first user was added succesfully, while on the second try with different data, exception was thrown (and caught).
In the console, where you ran the server script you will see:
home-debian:~/www/thrift-test/server# ./python_server.py Starting the server... ping() Processing user John Smith [User(user_id=1, description=None, firstname='John', lastname='Smith', sex=1, active=True)] Clearing list [User(user_id=1, description=None, firstname='John', lastname='Smith', sex=1, active=True)] []
Congratulations! You ran your first Thrift-enabled application!
I will be glad to hear from you about how do you like my tutorial. Please, leave a comment here(http://www.thrift.pl/Thrift-tutorial-comments.html).
