ActiveMQ多节点集群-基于zookeeper + replicated-leveldb-store的主从集群

一、原理说明

官网地址:http://activemq.apache.org/replicated-leveldb-store

原理图:

image

原理说明:

1)使用Zookeeper集群注册所有的ActiveMQ Broker,但只有其中一个Broker可以提供服务,它将被视为Master,其他的Broker处于待机状态被视为Slave。如果Master因故障而不能提供服务,Zookeeper会从Slave中选举出一个Broker充当Master。

2)Slave连接Master并同步他们的存储状态,Slave不接受客户端连接。所有的存储操作都将被复制到连接至Maste的Slaves。

3)如果Master宕机得到了最新更新的Slave会变成Master。故障节点在恢复后会重新加入到集群中并连接Master进入Slave模式。
 
4)所有需要同步的消息操作都将等待存储状态被复制到其他法定节点的操作完成才能完成。所以,如给你配置了replicas=3,name法定大小是(3/2)+1 = 2。Master将会存储更新然后等待(2-1)=1个Slave存储和更新完成,才汇报success。有一个node要作为观察者存在。当一个新的Master被选中,你需要至少保障一个法定mode在线以能够找到拥有最新状态的ode,这个node才可以成为新的Master。因此,推荐运行至少3个replica nodes以防止一个node失败后服务中断。

二、集群规划

主机 zookeeper集群端口 zookeeper安装目录 AMQ集群bind端口 AMQ消息tcp端口 AMQ管理控制台端口 AMQ安装目录
10.0.0.11 2181 /data/app/zookeeper bind=”tcp://0.0.0.0:63631” 61616 8161 /data/app/activemq
10.0.0.12 2181 /data/app/zookeeper bind=”tcp://0.0.0.0:63632” 61616 8162 /data/app/activemq
10.0.0.13 2181 /data/app/zookeeper bind=”tcp://0.0.0.0:63633” 61616 8163 /data/app/activemq

三、zookeeper集群

参考文档:https://www.cnblogs.com/hujinzhong/p/11825535.html

1)安装jdk8

[root@activemq01 ~]# rpm -ivh jdk-8u181-linux-x64.rpm
[root@activemq01 ~]# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

2)解压zookeeper安装包至指定目录

[root@activemq01 ~]# tar xf zookeeper-3.3.6.tar.gz -C /data/app/
[root@activemq01 ~]# mv /data/app/zookeeper-3.3.6/ /data/app/zookeeper

3)配置文件修改

[root@activemq01 conf]# pwd
/data/app/zookeeper/conf
[root@activemq01 conf]# cp zoo_sample.cfg zoo.cfg
[root@activemq01 conf]# vim zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/opt/zookeeper
# the port at which the clients will connect
clientPort=2181

#创建数据目录
[root@activemq01 conf]# mkdir -p /opt/zookeeper

4)启动zookeeper并查看状态

[root@activemq01 ~]# /data/app/zookeeper/bin/zkServer.sh start
JMX enabled by default
Using config: /data/app/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@activemq01 ~]# jps
17477 QuorumPeerMain
17498 Jps

[root@activemq01 ~]# /data/app/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /data/app/zookeeper/bin/../conf/zoo.cfg
Mode: standalone
[root@activemq01 ~]# netstat -lntp|grep 2181
tcp6       0      0 :::2181                 :::*                    LISTEN      17477/java

5)其他两台做相同配置

[root@activemq02 conf]# ../bin/zkServer.sh status
JMX enabled by default
Using config: /data/app/zookeeper/bin/../conf/zoo.cfg
Mode: standalone
[root@activemq02 conf]# netstat -lntp|grep 2181
tcp6       0      0 :::2181                 :::*                    LISTEN      16530/java

[root@active03 conf]# ../bin/zkServer.sh status
JMX enabled by default
Using config: /data/app/zookeeper/bin/../conf/zoo.cfg
Mode: standalone
[root@active03 conf]# netstat -lntp|grep 2181
tcp6       0      0 :::2181                 :::*                    LISTEN      17204/java

6)创建myid文件

#三台不一样
[root@activemq01 ~]# echo 1 > /opt/zookeeper/myid
[root@activemq02 ~]# echo 2 > /opt/zookeeper/myid
[root@active03 ~]# echo 3 > /opt/zookeeper/myid

7)修改配置文件

#三台一样
[root@activemq01 ~]# vim /data/app/zookeeper/conf/zoo.cfg 
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/opt/zookeeper
# the port at which the clients will connect
clientPort=2181

#cluster
server.1=10.0.0.11:2888:3888
server.2=10.0.0.12:2888:3888
server.3=10.0.0.13:2888:3888

8)重启三台zookeeper

[root@activemq01 ~]# /data/app/zookeeper/bin/zkServer.sh restart
[root@activemq02 ~]# /data/app/zookeeper/bin/zkServer.sh restart
[root@activemq03 ~]# /data/app/zookeeper/bin/zkServer.sh restart

#查看三台的状态
[root@activemq01 ~]# /data/app/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /data/app/zookeeper/bin/../conf/zoo.cfg
Mode: leader

[root@activemq02 ~]# /data/app/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /data/app/zookeeper/bin/../conf/zoo.cfg
Mode: follower

[root@active03 ~]# /data/app/zookeeper/bin/zkServer.sh status
JMX enabled by default
Using config: /data/app/zookeeper/bin/../conf/zoo.cfg
Mode: follower

四、activemq集群

1)下载并解压activemq(三台相同)

[root@activemq01 ~]# tar xf apache-activemq-5.15.9-bin.tar.gz -C /data/app/
[root@activemq01 ~]# mv /data/app/apache-activemq-5.15.9/ /data/app/activemq

2)修改管理控制台端口

[root@activemq01 ~]# cd /data/app/activemq/conf/
[root@activemq01 conf]# vim jetty.xml
109         <property name="host" value="0.0.0.0"/>
110         <property name="port" value="8161"/>

activemq01配置:

image

activemq02配置:

image

activemq03配置:

image

3)3个节点的brokerName一致(activemq.xml)

image

4)持久化配置

<persistenceAdapter>
   <replicatedLevelDB
		directory="${activemq.data}/leveldb"
		replicas="3"
		bind="tcp://0.0.0.0:63631"
		zkAddress="10.0.0.11:2181,10.0.0.12:2181,10.0.0.13:2181"
		hostname="10.0.0.11"
		zkPath="/activemq/leveldb-stores"
		/>
</persistenceAdapter>

activemq01配置:

image

activemq02配置:

image

activemq03配置:

image

5)启动activemq集群

前提:先启动zookeeper集群

[root@activemq01 activemq]# ./bin/activemq start
[root@activemq02 activemq]# ./bin/activemq start
[root@activemq03 activemq]# ./bin/activemq start

#activemq01
[root@activemq01 app]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1134/sshd           
tcp6       0      0 :::61613                :::*                    LISTEN      20689/java          
tcp6       0      0 :::61614                :::*                    LISTEN      20689/java          
tcp6       0      0 :::63631                :::*                    LISTEN      20689/java          
tcp6       0      0 :::61616                :::*                    LISTEN      20689/java          
tcp6       0      0 :::3888                 :::*                    LISTEN      18386/java          
tcp6       0      0 :::42611                :::*                    LISTEN      20689/java          
tcp6       0      0 :::22                   :::*                    LISTEN      1134/sshd           
tcp6       0      0 :::1883                 :::*                    LISTEN      20689/java          
tcp6       0      0 :::8161                 :::*                    LISTEN      20689/java          
tcp6       0      0 :::33250                :::*                    LISTEN      18386/java          
tcp6       0      0 :::2181                 :::*                    LISTEN      18386/java          
tcp6       0      0 :::5672                 :::*                    LISTEN      20689/java          
tcp6       0      0 :::2888                 :::*                    LISTEN      18386/java

#activemq02
[root@activemq02 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1091/sshd           
tcp6       0      0 :::3888                 :::*                    LISTEN      17230/java          
tcp6       0      0 :::35284                :::*                    LISTEN      19503/java          
tcp6       0      0 :::22                   :::*                    LISTEN      1091/sshd           
tcp6       0      0 :::40127                :::*                    LISTEN      17230/java          
tcp6       0      0 :::2181                 :::*                    LISTEN      17230/java

#activemq03
[root@activemq03 ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1089/sshd           
tcp6       0      0 :::3888                 :::*                    LISTEN      17812/java          
tcp6       0      0 :::22                   :::*                    LISTEN      1089/sshd           
tcp6       0      0 :::40284                :::*                    LISTEN      17812/java          
tcp6       0      0 :::40066                :::*                    LISTEN      20141/java          
tcp6       0      0 :::2181                 :::*                    LISTEN      17812/java 

6)登录zookeeper查看activemq集群状态

[root@activemq01 ~]# /data/app/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181
[zk: 127.0.0.1:2181(CONNECTED) 0] ls /
[activemq, zookeeper]
[zk: 127.0.0.1:2181(CONNECTED) 1] ls /activemq
[leveldb-stores]
[zk: 127.0.0.1:2181(CONNECTED) 2] ls /activemq/leveldb-stores
[00000000002, 00000000000, 00000000001]
[zk: 127.0.0.1:2181(CONNECTED) 3] get /activemq/leveldb-stores/0000000000
00000000002   00000000000   00000000001
[zk: 127.0.0.1:2181(CONNECTED) 3] get /activemq/leveldb-stores/00000000000
{"id":"activemq","container":null,"address":"tcp://10.0.0.11:63631","position":-1,"weight":1,"elected":"0000000000"}  #master
cZxid = 0x200000004
ctime = Sat Mar 21 14:43:41 CST 2020
mZxid = 0x20000000c
mtime = Sat Mar 21 14:44:19 CST 2020
pZxid = 0x200000004
cversion = 0
dataVersion = 4
aclVersion = 0
ephemeralOwner = 0x170fbadab630000
dataLength = 116
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 4] get /activemq/leveldb-stores/00000000001
{"id":"activemq","container":null,"address":null,"position":-1,"weight":1,"elected":null} #slave
cZxid = 0x200000007
ctime = Sat Mar 21 14:44:19 CST 2020
mZxid = 0x20000000a
mtime = Sat Mar 21 14:44:19 CST 2020
pZxid = 0x200000007
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x170fbadab630001
dataLength = 89
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 5] get /activemq/leveldb-stores/00000000002  #slave
{"id":"activemq","container":null,"address":null,"position":-1,"weight":1,"elected":null}
cZxid = 0x20000000e
ctime = Sat Mar 21 14:44:31 CST 2020
mZxid = 0x20000000e
mtime = Sat Mar 21 14:44:31 CST 2020
pZxid = 0x20000000e
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x170fbadab630002
dataLength = 89
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 6] 

6)管理界面访问

此时只能访问master的管理界面,slave的不能提供访问

image

五、activemq测试故障切换

#关闭master
[root@activemq01 ~]# /data/app/activemq/bin/activemq stop

#查看选举状态
[zk: 127.0.0.1:2181(CONNECTED) 0] ls /
[activemq, zookeeper]
[zk: 127.0.0.1:2181(CONNECTED) 1] ls /activemq
[leveldb-stores]
[zk: 127.0.0.1:2181(CONNECTED) 2] ls /activemq/leveldb-stores
[00000000002, 00000000001]
[zk: 127.0.0.1:2181(CONNECTED) 4] get /activemq/leveldb-stores/00000000001
{"id":"activemq","container":null,"address":"tcp://10.0.0.12:63632","position":-1,"weight":1,"elected":"0000000001"} #master
cZxid = 0x200000007
ctime = Sat Mar 21 14:44:19 CST 2020
mZxid = 0x20000001a
mtime = Sat Mar 21 14:57:44 CST 2020
pZxid = 0x200000007
cversion = 0
dataVersion = 6
aclVersion = 0
ephemeralOwner = 0x170fbadab630001
dataLength = 116
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 5] get /activemq/leveldb-stores/00000000002
{"id":"activemq","container":null,"address":null,"position":-1,"weight":1,"elected":null}
cZxid = 0x20000000e
ctime = Sat Mar 21 14:44:31 CST 2020
mZxid = 0x200000017
mtime = Sat Mar 21 14:57:43 CST 2020
pZxid = 0x20000000e
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x170fbadab630002
dataLength = 89
numChildren = 0

此时activemq02成为master,登录管理界面测试

image

重新启动activemq01,将加入集群成为slave

posted @ 2020-03-21 13:05  运维人在路上  阅读(705)  评论(0编辑  收藏  举报