rocketmq 记

Rocketmq选型

Rocket是一个专业的队列服务,性能优于Rabbitmq,优势是性能和并发,源于Kafka的扩展版,增强了数据的可靠性。

Rocketmq的队列类型

普通队列,广播队列、顺序队列,分区顺序

2、同步机制

Rocketmq使用主从同步模式,同步分为同步和异步模式,这和mysql类似。

3、Rocketmq管理命令

rocketmq也可以通过web管理,坑中有说 

创建topic

bin/mqadmin updateTopic -n '192.168.1.64:9876;192.168.1.107:9876' -c DefaultCluster -t tt -w 5 -r 5 -o=true   #-o=true是有序队列

Rocketmq坑

1、rocketmq 在本地搭建环境启动时,内存溢出报了内存错误:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.

解决方法:修改bin/runbroker.sh,bin/runserver.sh

-server -Xms1g -Xmx1g -Xmn1g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m

内存大的虚拟机没问题,像我这渣渣笔记本,得节省点啊!

 

2、Rocketmq管理web

https://github.com/apache/rocketmq-externals/tree/master/rocketmq-console

下载安装Console项目

 

Rocketmq的普通队列、顺序队列、分区顺序队列、广播消息实现、事务队列

 

https://www.jianshu.com/p/3afd610a8f7d

https://www.jianshu.com/p/790d6bc4a1c1

https://www.jianshu.com/p/53324ea2df92

 

我遇到的坑

阿里云上面买的RocketMq和开源版本的Rocketmq的sdk不同,有着兼容问题。我们有一个自己机房和阿里的共用运行的服务,这时候发现只能在阿里机房上布,而在本地机房看来只能再写一套抽象层了,这非常不友好。使用了阿里的 Rocketmq也就意味着你的程序移植不到其它云或服务器上了。

slave 修改:

1、内存优化

bin/runbroker.sh MaxDirectMemorySize =15g  改成1g

 

2、host解析失败

在rocketmq的主从是通过获取hostname然后解析host到ip上进行互相访问的,所以需要将各机器的hosts增加解析

 

192.168.1.102       localhost.ceontsPHP

192.168.1.101       localhost.bobby

 

hostname可以通过各服务器的/etc/hostsname查看

rocket会报错:rocketmq Failed to obtain the host name

 

 

3、锁文件冲突原因

因为rocketmq设计一个用户运行一个示例,一个用户锁定了一个用户store存储目录,所以在一起运行会冲突,提示:Lock failed,MQ already startedz

解决方法是每个示例使用一个用户

broker.conf 指定 storePathRootDir目录

 

 

4、端口会冲突

不同broker需使用不同端口,如果不是root用户,端口号应该是1024+

一个broker不仅仅使用你指定的listenPort,上下最好空几个,比如指定1081端口,他还占用了1079 1081 1082

 

5、配置文件坑

我直接拿官方配置文件2m-2s-async下的配置,不行,没有提示,就是执行clusterList时,没有集群,这很坑,然后我自己建配置文件成功

 

bin/mqbroker -c /root/rocketmq/release/conf/2m-2s-async/broker-a-s.properties > a.log 2>&1

 

bin/mqadmin clusterList -n centos:9876

 

bin/mqadmin updateTopic -n 'centos:9876' -c mycluster -t tt -w 5 -r 5 -o=true

 

# broker a

namesrvAddr=centos:9876

brokerIP1=192.168.56.101

brokerName=broker-a

brokerClusterName=mycluster

brokerId=0

listenPort=1081

autoCreateTopicEnable=true

autoCreateSubscriptionGroup=true

rejectTransactionMessage=false

fetchNamesrvAddrByAddressServer=false

storePathRootDir=/root/storeA

storePathCommitLog=/root/store/commitlog

flushIntervalCommitLog=500

commitIntervalCommitLog=200

flushCommitLogTimed=false

deleteWhen=04

fileReservedTime=48

maxTransferBytesOnMessageInMemory=262144

maxTransferCountOnMessageInMemory=32

maxTransferBytesOnMessageInDisk=65536

maxTransferCountOnMessageInDisk=8

accessMessageInMemoryMaxRatio=40

messageIndexEnable=true

messageIndexSafe=false

haMasterAddress=

brokerRole=ASYNC_MASTER

flushDiskType=ASYNC_FLUSH

cleanFileForciblyEnable=true

transientStorePoolEnable=false

#broker a-s

 

namesrvAddr=centos:9876

brokerIP1=192.168.56.102

brokerName=broker-a

brokerClusterName=mycluster

brokerId=1

listenPort=1091

autoCreateTopicEnable=true

autoCreateSubscriptionGroup=true

rejectTransactionMessage=false

fetchNamesrvAddrByAddressServer=false

storePathRootDir=/root/storeAs

storePathCommitLog=/root/store/commitlog

flushIntervalCommitLog=500

commitIntervalCommitLog=200

flushCommitLogTimed=false

deleteWhen=04

fileReservedTime=48

maxTransferBytesOnMessageInMemory=262144

maxTransferCountOnMessageInMemory=32

maxTransferBytesOnMessageInDisk=65536

maxTransferCountOnMessageInDisk=8

accessMessageInMemoryMaxRatio=40

messageIndexEnable=true

messageIndexSafe=false

haMasterAddress=

brokerRole=SLAVE

flushDiskType=ASYNC_FLUSH

cleanFileForciblyEnable=true

transientStorePoolEnable=false

 

#broker b

namesrvAddr=centos:9876

brokerIP1=192.168.56.101

brokerName=broker-b

brokerClusterName=mycluster

brokerId=0

listenPort=1085

autoCreateTopicEnable=true

autoCreateSubscriptionGroup=true

rejectTransactionMessage=false

fetchNamesrvAddrByAddressServer=false

storePathRootDir=/root/storeB

storePathCommitLog=/root/store/commitlog

flushIntervalCommitLog=500

commitIntervalCommitLog=200

flushCommitLogTimed=false

deleteWhen=04

fileReservedTime=48

maxTransferBytesOnMessageInMemory=262144

maxTransferCountOnMessageInMemory=32

maxTransferBytesOnMessageInDisk=65536

maxTransferCountOnMessageInDisk=8

accessMessageInMemoryMaxRatio=40

messageIndexEnable=true

messageIndexSafe=false

haMasterAddress=

brokerRole=ASYNC_MASTER

flushDiskType=ASYNC_FLUSH

cleanFileForciblyEnable=true

transientStorePoolEnable=false

#broker b-s

namesrvAddr=centos:9876

brokerIP1=192.168.56.102

brokerName=broker-b

brokerClusterName=mycluster

brokerId=1

listenPort=1095

autoCreateTopicEnable=true

autoCreateSubscriptionGroup=true

rejectTransactionMessage=false

fetchNamesrvAddrByAddressServer=false

storePathRootDir=/root/storeBs

storePathCommitLog=/root/store/commitlog

flushIntervalCommitLog=500

commitIntervalCommitLog=200

flushCommitLogTimed=false

deleteWhen=04

fileReservedTime=48

maxTransferBytesOnMessageInMemory=262144

maxTransferCountOnMessageInMemory=32

maxTransferBytesOnMessageInDisk=65536

maxTransferCountOnMessageInDisk=8

accessMessageInMemoryMaxRatio=40

messageIndexEnable=true

messageIndexSafe=false

haMasterAddress=

brokerRole=SLAVE

flushDiskType=ASYNC_FLUSH

cleanFileForciblyEnable=true

transientStorePoolEnable=false

 

rocketmq sdk使用newFixThreadPool线程池用于获取server中的数据(线程数获取的cpu核心数),拿到消息后,submit 给->ThreadPoolExecutor线程池进行执行消费,执行消费可以指定最大线程数和最小线程数。

rocketmq多消费端是需要通过tag指定的,如果你想和使用rabbitmq一样使用,可能会疯,百度搜出来的鸡八狗屁文章,根本狗屁不通。

 

rocketmq 存储结构:
https://blog.csdn.net/mr253727942/article/details/55805876

IO调度方式:

https://www.cnblogs.com/cobbliu/p/5389556.html

 

posted @ 2018-08-06 22:09  阿旭^_^  阅读(1884)  评论(0编辑  收藏  举报