Zookeeper笔记之使用zk实现集群选主

 

一、需求

在主从结构的集群中，我们假设硬件机器是很脆弱的，随时可能会宕机，当master挂掉之后需要从slave中选出一个节点作为新的master，使用zookeeper可以很简单的实现集群选主功能。

 

二、分析

下面为了方便叙述，将使用更通用的技术术语，即使用leader表示master，使用follower表示slave。

集群选主涉及到两个问题：

1. 谁来做leader

2. leader挂掉了怎么被follower感知到

首先是第一个问题，谁来做leader，其实可以将这个问题看做是多线程中的互斥锁抢占，锁只有一把，并且只能被一个人抢到，这里就把一个zookeeper上的一个节点/leader-info看做是锁，集群中的每台机器都尝试去创建这个节点（抢占锁），因为zookeeper创建节点是原子性操作，所以只有一台机器能够创建成功其它都会失败，创建成功的那台机器就作为leader，其它机器做follower，一般leader抢占成功了之后会在/leader-info节点上存储一些与自己相关的信息，比如hostname、id之类的，以让follower知道谁抢占成功成为了leader，然后去连接leader进行一些数据交换或指令控制之类的，那就是选主之后的事了不在此篇文章的讨论范围之内。

第二个问题是leader挂掉了怎么通知其它的follower，zookeeper中的节点按照有效时间分为持久节点和临时节点，临时节点跟session绑定，而一个session表示一个客户端，当客户端下线的时候session失效，当session失效的时候跟它绑定的临时节点就会被删除，利用这个特性可以检测节点是否还在存活状态，实现follower对leader下线的感知，只是需要注意在创建/leader-info节点的时候要将其创建为临时节点，然后众多follower都在这个节点上添加一个watcher监听其删除事件，这样当leader挂掉的时候session失效，然后与此session绑定的临时节点会被删除，即/leader-info节点将被删除，同时给所有的follower发送事件通知，follower一看leader挂了就燥起来了，将自己的状态置为looking，开始新一轮的选举。

 

总结一下选主的流程：

1. 集群中的所有机器将自己置为looking状态，准备开始选举。

2. 所有looking状态的机器尝试去创建/leader-info节点。

3. 创建成功的将自己的状态修改为leader，同时将自己的一些信息写入到/leader-info这个节点上。

4. 创建失败的将自己的状态置为follower，同时尝试从/leader-info获取leader信息进行一些leader改变的逻辑（在这里这个不是重点，打印一下即可），follower在获取/leader-info节点数据的同时要设置一个watcher，监听此节点的删除事件，当节点被删除事件触发时启动新一轮的选举，因为获取数据设置watcher这个操作是原子性的，所以要么这个节点存在获取数据成功，并且设置watcher也成功，要么节点不存在抛出KeeperException.NoNodeException异常。

5. 为什么在follower设置watcher的时候还有可能会抛异常呢，leader不是已经创建了这个节点了吗？因为follower从尝试创建/leader-info节点失败到去获取此节点的数据同时设置watcher这一段操作不是原子性的，在这中间可能会发生一些变故，leader可能刚成为leader就挂掉了（或者因为一些网络抖动原因，总之是session失效了），leader挂掉之后它创建的临时节点就被zookeeper删除了，所以当follower在设置watcher的时候如果检测到KeeperException.NoNodeException，说明之前的leader挂掉了，此时集群中已经没有了leader，follower又燥起来了，它将自己的状态置为looking开始新一轮的选举。

 

三、实现

Node.java：

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

package cc11001100.zookeeper.leaderElection;   import cc11001100.zookeeper.utils.ZooKeeperUtil; import org.apache.zookeeper.CreateMode; import org.apache.zookeeper.KeeperException; import org.apache.zookeeper.Watcher; import org.apache.zookeeper.ZooDefs; import org.apache.zookeeper.ZooKeeper;   import java.io.IOException; import java.io.UnsupportedEncodingException;   /**  * 表示集群中的一个节点  *  * @author CC11001100  */ public class Node {       private Status status;     private String nodeForLeaderInfo;     private ZooKeeper zooKeeper;       public Node(String listenerNodeForLeader) throws IOException {         this.nodeForLeaderInfo = listenerNodeForLeader;         this.zooKeeper = ZooKeeperUtil.getZooKeeper();         lookingForLeader();     }       public void lookingForLeader() {         status = Status.LOOKING;         try {             String leaderInfo = Thread.currentThread().getName();             // 需要注意这里创建的是临时节点             zooKeeper.create(nodeForLeaderInfo, leaderInfo.getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);             // 如果上一步没有抛异常，说明自己已经是leader了             status = Status.LEADER;             String logMsg = Thread.currentThread().getName() + " is leader";             System.out.println(logMsg);         } catch (KeeperException.NodeExistsException e) {             // 节点已经存在，说明leader已经被别人注册成功了，自己是follower             status = Status.FOLLOWER;             try {                 byte[] leaderInfoBytes = zooKeeper.getData(nodeForLeaderInfo, event -> {                     if (event.getType() == Watcher.Event.EventType.NodeDeleted) {                         lookingForLeader();                     }                 }, null);                 String logMsg = Thread.currentThread().getName() + " is follower, master is " + new String(leaderInfoBytes, "UTF-8");                 System.out.println(logMsg);             } catch (KeeperException.NoNodeException e1) {                 // 如果在获取leader信息的时候报了节点不存在，说明这个leader比较短命，刚抢到leader就又挂掉了                 lookingForLeader();             } catch (KeeperException | InterruptedException | UnsupportedEncodingException e1) {                 e1.printStackTrace();             }         } catch (KeeperException | InterruptedException e) {             e.printStackTrace();         }     }       public void shutdown() {         try {             if (zooKeeper != null) {                 zooKeeper.close();             }         } catch (InterruptedException e) {             e.printStackTrace();         }     }       public Status getStatus() {         return status;     }       // 当前节点的状态，节点的状态必须在这三个中的一个     public enum Status {         LOOKING, // 选举中         LEADER, // 选举完毕，当前节点为leader         FOLLOWER; // 选举完毕，当前节点为follower     }   }

LeaderElectionTest.java：

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57

package cc11001100.zookeeper.leaderElection;   import java.io.IOException; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong;   /**  * @author CC11001100  */ public class LeaderElectionTest {       private static void sleep(long mils) {         try {             TimeUnit.MILLISECONDS.sleep(mils);         } catch (InterruptedException e) {             e.printStackTrace();         }     }       public static void main(String[] args) throws IOException {           final String LEADER_INFO_NODE = "/leader-info";         int nodeNum = 10;         AtomicLong idGenerator = new AtomicLong();         AtomicInteger activeNodeCount = new AtomicInteger();         while (true) {             if (activeNodeCount.get() >= nodeNum) {                 sleep(10);                 continue;             }               // 线程启动需要一定时间，将线程启动看做开机过程，在开机之前就算一台新的机器加入了             activeNodeCount.incrementAndGet();             new Thread(() -> {                 try {                     Node node = new Node(LEADER_INFO_NODE);                     while (true) {                         sleep(1000);                         // 这里为了试验就让leader有轻微自杀倾向...                         if (node.getStatus() == Node.Status.LEADER && Math.random() < 0.3) {                             String logMsg = "----------------------------- " + Thread.currentThread().getName() + " shutdown -----------------------------";                             System.out.println(logMsg);                             node.shutdown();                             break;                         }                     }                 } catch (IOException e) {                     e.printStackTrace();                 } finally {                     activeNodeCount.decrementAndGet();                 }             }, "node-" + idGenerator.getAndIncrement()).start();         }     }   }

控制台输出：

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

... node-4 is leader node-3 is follower, master is node-4 node-0 is follower, master is node-4 node-9 is follower, master is node-4 node-7 is follower, master is node-4 node-5 is follower, master is node-4 node-1 is follower, master is node-4 node-6 is follower, master is node-4 node-8 is follower, master is node-4 node-2 is follower, master is node-4 ----------------------------- node-4 shutdown ----------------------------- node-0-EventThread is leader node-6-EventThread is follower, master is node-0-EventThread node-3-EventThread is follower, master is node-0-EventThread node-7-EventThread is follower, master is node-0-EventThread node-1-EventThread is follower, master is node-0-EventThread node-5-EventThread is follower, master is node-0-EventThread node-9-EventThread is follower, master is node-0-EventThread node-2-EventThread is follower, master is node-0-EventThread node-8-EventThread is follower, master is node-0-EventThread node-10 is follower, master is node-0-EventThread ----------------------------- node-0 shutdown ----------------------------- node-6-EventThread is leader node-7-EventThread is follower, master is node-6-EventThread node-1-EventThread is follower, master is node-6-EventThread node-3-EventThread is follower, master is node-6-EventThread node-10-EventThread is follower, master is node-6-EventThread node-9-EventThread is follower, master is node-6-EventThread node-5-EventThread is follower, master is node-6-EventThread node-2-EventThread is follower, master is node-6-EventThread node-8-EventThread is follower, master is node-6-EventThread node-11 is follower, master is node-6-EventThread ----------------------------- node-6 shutdown ----------------------------- node-1-EventThread is leader node-10-EventThread is follower, master is node-1-EventThread node-7-EventThread is follower, master is node-1-EventThread node-11-EventThread is follower, master is node-1-EventThread node-8-EventThread is follower, master is node-1-EventThread node-5-EventThread is follower, master is node-1-EventThread node-9-EventThread is follower, master is node-1-EventThread node-3-EventThread is follower, master is node-1-EventThread node-2-EventThread is follower, master is node-1-EventThread node-12 is follower, master is node-1-EventThread ----------------------------- node-1 shutdown ----------------------------- node-3-EventThread is leader node-12-EventThread is follower, master is node-3-EventThread node-11-EventThread is follower, master is node-3-EventThread node-5-EventThread is follower, master is node-3-EventThread node-7-EventThread is follower, master is node-3-EventThread node-9-EventThread is follower, master is node-3-EventThread node-2-EventThread is follower, master is node-3-EventThread node-10-EventThread is follower, master is node-3-EventThread node-8-EventThread is follower, master is node-3-EventThread node-13 is follower, master is node-3-EventThread ----------------------------- node-3 shutdown ----------------------------- node-5-EventThread is leader node-13-EventThread is follower, master is node-5-EventThread node-12-EventThread is follower, master is node-5-EventThread node-7-EventThread is follower, master is node-5-EventThread node-11-EventThread is follower, master is node-5-EventThread node-10-EventThread is follower, master is node-5-EventThread node-9-EventThread is follower, master is node-5-EventThread node-2-EventThread is follower, master is node-5-EventThread node-8-EventThread is follower, master is node-5-EventThread node-14 is follower, master is node-5-EventThread ----------------------------- node-5 shutdown ----------------------------- node-7-EventThread is leader node-13-EventThread is follower, master is node-7-EventThread node-12-EventThread is follower, master is node-7-EventThread node-9-EventThread is follower, master is node-7-EventThread node-11-EventThread is follower, master is node-7-EventThread node-14-EventThread is follower, master is node-7-EventThread node-10-EventThread is follower, master is node-7-EventThread node-8-EventThread is follower, master is node-7-EventThread node-2-EventThread is follower, master is node-7-EventThread node-15 is follower, master is node-7-EventThread ----------------------------- node-7 shutdown ----------------------------- node-14-EventThread is leader node-13-EventThread is follower, master is node-14-EventThread node-11-EventThread is follower, master is node-14-EventThread node-2-EventThread is follower, master is node-14-EventThread node-12-EventThread is follower, master is node-14-EventThread node-15-EventThread is follower, master is node-14-EventThread node-10-EventThread is follower, master is node-14-EventThread node-9-EventThread is follower, master is node-14-EventThread node-8-EventThread is follower, master is node-14-EventThread node-16 is follower, master is node-14-EventThread ----------------------------- node-14 shutdown ----------------------------- node-13-EventThread is leader node-12-EventThread is follower, master is node-13-EventThread node-15-EventThread is follower, master is node-13-EventThread node-9-EventThread is follower, master is node-13-EventThread node-10-EventThread is follower, master is node-13-EventThread node-2-EventThread is follower, master is node-13-EventThread node-8-EventThread is follower, master is node-13-EventThread node-11-EventThread is follower, master is node-13-EventThread node-16-EventThread is follower, master is node-13-EventThread node-17 is follower, master is node-13-EventThread ...

 

最后有个需要注意的地方就是是否需要将leader节点设置为顺序临时节点呢？类似于分布式锁那样，这样的话每次唤醒一个节点就可以了，这看上去像是一个可以优化的点。

其实并不是，当leader挂掉的时候必须所有follower都能够感知到leader的变更，即使他们不参与抢主也必须醒来执行leader变更的逻辑。

 

.