Zookeeper和zkpython
简介:
Zookeeper (http://zookeeper.apache.org/)分布式服务框架是 Apache Hadoop 的一个子项目,它主要是用来解决分布式应用中经常遇到的一些数据管理问题,如:统一命名服务、状态同步服务、集群管理、分布式应用配置项的管理等。
具体简介可以参照这篇文章。
zkpython的安装:
python中有一个zkpython的包,是基于zookeeper的c-client开发的,所以安装的时候需要先安装zookeeper的c客户端。安装步骤如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# 首先下载zookeeper wget http: //labs .renren.com /apache-mirror//zookeeper/zookeeper-3 .3.3 /zookeeper-3 .3.3. tar .gz tar xzvf zookeeper-3.3.3. tar .gz cd zookeeper-3.3.3 /src/c/ . /configure make make install # 然后下载zkpython wget http: //pypi .python.org /packages/source/z/zkpython/zkpython-0 .4. tar .gz #md5=3de220615aaddf57f1462b78d32477f9 tar xzvf zkpython-0.4. tar .gz cd zkpython-0.4 python setup.py install |
这样就完成了zkpython的安装。
一个简单的demo:
之后让我们来写一个简单的demo吧。(demo中用到的zkclient.py:https://github.com/piglei/zkpython_example/blob/master/zkclient.py)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
|
# coding: utf-8 import logging from os.path import basename, join from zkclient import ZKClient, zookeeper, watchmethod logging.basicConfig( level = logging.DEBUG, format = "[%(asctime)s] %(levelname)-8s %(message)s" ) log = logging class GJZookeeper( object ): ZK_HOST = "localhost:2181" ROOT = "/app" WORKERS_PATH = join(ROOT, "workers" ) MASTERS_NUM = 1 TIMEOUT = 10000 def __init__( self , verbose = True ): self .VERBOSE = verbose self .masters = [] self .is_master = False self .path = None self .zk = ZKClient( self .ZK_HOST, timeout = self .TIMEOUT) self .say( "login ok!" ) # init self .__init_zk() # register self .register() def __init_zk( self ): """ create the zookeeper node if not exist """ nodes = ( self .ROOT, self .WORKERS_PATH) for node in nodes: if not self .zk.exists(node): try : self .zk.create(node, "") except : pass @property def is_slave( self ): return not self .is_master def register( self ): """ register a node for this worker """ self .path = self .zk.create( self .WORKERS_PATH + "/worker" , "1" , flags = zookeeper.EPHEMERAL | zookeeper.SEQUENCE) self .path = basename( self .path) self .say( "register ok! I'm %s" % self .path) # check who is the master self .get_master() def get_master( self ): """ get children, and check who is the smallest child """ @watchmethod def watcher(event): self .say( "child changed, try to get master again." ) self .get_master() children = self .zk.get_children( self .WORKERS_PATH, watcher) children.sort() self .say( "%s's children: %s" % ( self .WORKERS_PATH, children)) # check if I'm master self .masters = children[: self .MASTERS_NUM] if self .path in self .masters: self .is_master = True self .say( "I've become master!" ) else : self .say( "%s is masters, I'm slave" % self .masters) def say( self , msg): """ print messages to screen """ if self .VERBOSE: if self .path: log.info( "[ %s(%s) ] %s" % ( self .path, "master" if self .is_master else "slave" , msg)) else : log.info(msg) def main(): gj_zookeeper = GJZookeeper() if __name__ = = "__main__" : main() import time time.sleep( 1000 ) |
这个简单的demo所做的事情,就是通过在zookeeper的/app/workers节点下建立临时的子节点( flags=zookeeper.EPHEMERAL | zookeeper.SEQUENCE ),每次create完成之后检查自己是不是在最小的MASTERS_NUM(例子中为1,即单master)里。如果是的话,作为master运行,否则的话,作为slave运行。
这样的话,当我们的master挂掉以后,与zookeeper之间的连接也会中断,过了指定的TIMEOUT以后,master之前在worker下的子节点就会被删除,于是slave节点之前设置的watcher会被触发,再次检查自己是否为master,如果是的话则完成切换。
demo运行结果:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# 第一个实例 Connected in 20 ms, handle is 0 [2011-09-09 12:40:43,702] INFO login ok! Node /app/workers/worker created in 4 ms [2011-09-09 12:40:43,708] INFO [ worker0000000022(slave) ] register ok! I'm worker0000000022 [2011-09-09 12:40:43,709] INFO [ worker0000000022(slave) ] /app/workers's children: ['worker0000000022'] [2011-09-09 12:40:43,709] INFO [ worker0000000022(master) ] I've become master! # 这时再起第二个实例 Connected in 64 ms, handle is 0 [2011-09-09 12:43:08,334] INFO login ok! Node /app/workers/worker created in 11 ms [2011-09-09 12:43:08,346] INFO [ worker0000000023(slave) ] register ok! I'm worker0000000023 [2011-09-09 12:43:08,347] INFO [ worker0000000023(slave) ] /app/workers's children: ['worker0000000022', 'worker0000000023'] [2011-09-09 12:43:08,347] INFO [ worker0000000023(slave) ] ['worker0000000022'] is masters, I'm slave # 杀掉master,第二个实例发生的变化 [2011-09-09 12:44:06,016] INFO [ worker0000000023(slave) ] child changed, try to get master again. [2011-09-09 12:44:06,017] INFO [ worker0000000023(slave) ] /app/workers's children: ['worker0000000023'] [2011-09-09 12:44:06,017] INFO [ worker0000000023(master) ] I've become master! |