Namesrv源码解析
NameSrvStartup 启动入口类
- 构建命令行参数 -c 配置文件 和 -p 打印配置项
- 将参数映射到NamesrvConfig和NettyServerConfig中，如listenPort可以配置启动端口
- 创建NamesrvController并启动
NamesrvController
NamesrvController 构造函数
public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig) {
this.namesrvConfig = namesrvConfig;
this.nettyServerConfig = nettyServerConfig;
this.kvConfigManager = new KVConfigManager(this);
this.routeInfoManager = new RouteInfoManager();
this.brokerHousekeepingService = new BrokerHousekeepingService(this);
this.configuration = new Configuration(
log,
this.namesrvConfig, this.nettyServerConfig
);
this.configuration.setStorePathFromConfig(this.namesrvConfig, "configStorePath");
}
- namesrvConfig 主要有rocketmqHome（rocketMq的Home目录）、kvConfigPath 等属性
- nettyServerConfig 远程通信的一些配置项 如listenPort 端口
- kvConfigManager kv配置管理器
- routeInfoManager 路由信息管理器
- brokerHousekeepingService 该类实现了ChannelEventListener接口，主要用于管理连接创建、关闭、销毁、异常时的一些处理
- configuration 将namesrvConfig与nettyServerConfig封装在这个配置类中
controller.initialize() 初始化
public boolean initialize() {
//根据kvConfigPath加载kv配置
this.kvConfigManager.load();
//创建一个netty远程服务器
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);
//默认处理器的线程池
this.remotingExecutor =
Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(), new ThreadFactoryImpl("RemotingExecutorThread_"));
//注册默认的请求处理器
this.registerProcessor();
//定时任务 scanNotActiveBroker 每隔10秒扫描剔除不活跃的broker
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.routeInfoManager.scanNotActiveBroker();
}
}, 5, 10, TimeUnit.SECONDS);
//定时任务 每隔10分钟打印全部kv信息
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
NamesrvController.this.kvConfigManager.printAllPeriodically();
}
}, 1, 10, TimeUnit.MINUTES);
return true;
}
1.根据kvConfigPath加载kv配置
2.创建一个netty远程服务器
3.创建默认处理器的线程池
4.注册默认的请求处理器
5.定时任务 scanNotActiveBroker 扫描剔除不活跃的broker
6.定时任务 每隔10分钟打印全部kv信息
controller.start() 启动
public void start() throws Exception {
//启动远程服务器
this.remotingServer.start();
}
1.调用remotingServer.start启动远程服务器，即NettyRemotingServer#start
remotingServer.start() 启动服务器
public void start() {
this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
nettyServerConfig.getServerWorkerThreads(),
new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
}
});
ServerBootstrap childHandler =
this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
.channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
.option(ChannelOption.SO_BACKLOG, 1024)
.option(ChannelOption.SO_REUSEADDR, true)
.option(ChannelOption.SO_KEEPALIVE, false)
.childOption(ChannelOption.TCP_NODELAY, true)
.childOption(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
.childOption(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
.localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline()
.addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME,
new HandshakeHandler(TlsSystemConfig.tlsMode))
.addLast(defaultEventExecutorGroup,
new NettyEncoder(),
new NettyDecoder(),
new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
new NettyConnectManageHandler(),
new NettyServerHandler()
);
}
});
if (nettyServerConfig.isServerPooledByteBufAllocatorEnable()) {
childHandler.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
}
try {
ChannelFuture sync = this.serverBootstrap.bind().sync();
InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
this.port = addr.getPort();
} catch (InterruptedException e1) {
throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
}
if (this.channelEventListener != null) {
this.nettyEventExecutor.start();
}
this.timer.scheduleAtFixedRate(new TimerTask() {
@Override
public void run() {
try {
NettyRemotingServer.this.scanResponseTable();
} catch (Throwable e) {
log.error("scanResponseTable exception", e);
}
}
}, 1000 * 3, 1000);
}
这里的代码就是netty构建服务器的代码，主要关注以下几点
- options参数
- ChannelOption.SO_BACKLOG 对应的是tcp/ip协议listen函数中的backlog参数，函数listen(int socketfd,int backlog)用来初始化服务端可连接队列，服务端处理客户端连接请求是顺序处理的，所以同一时间只能处理一个客户端连接，多个客户端来的时候，服务端将不能处理的客户端连接请求放在队列中等待处理，backlog参数指定了队列的大小
- ChannelOption.SO_REUSEADDR 一般来说，一个端口释放后会等待两分钟之后才能再被使用，SO_REUSEADDR是让端口释放后立即就可以被再次使用
- ChannelOption.SO_KEEPALIVE 当设置为true的时候，TCP会实现监控连接是否有效，当连接处于空闲状态的时候，超过了2个小时，本地的TCP实现会发送一个数据包给远程的 socket，如果远程没有发回响应，TCP会持续尝试11分钟，知道响应为止，如果在12分钟的时候还没响应，TCP尝试关闭socket连接。
- ChannelOption.TCP_NODELAY ChannelOption.TCP_NODELAY参数对应于套接字选项中的TCP_NODELAY,该参数的使用与Nagle算法有关,Nagle算法是将小的数据包组装为更大的帧然后进行发送，而不是输入一次发送一次,因此在数据包不足的时候会等待其他数据的到了，组装成大的数据包进行发送，虽然该方式有效提高网络的有效负载，但是却造成了延时，而该参数的作用就是禁止使用Nagle算法，使用于小数据即时传输，于TCP_NODELAY相对应的是TCP_CORK，该选项是需要等到发送的数据量最大的时候，一次性发送数据，适用于文件传输。
- ChannelOption.SO_SNDBUF 发送缓冲区大小
- ChannelOption.SO_RCVBUF 接收缓冲区大小
- handler 处理器
- NettyServerHandler 核心处理器 处理业务请求
NettyServerHandler
public void processMessageReceived(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
final RemotingCommand cmd = msg;
if (cmd != null) {
switch (cmd.getType()) {
case REQUEST_COMMAND:
processRequestCommand(ctx, cmd);
break;
case RESPONSE_COMMAND:
processResponseCommand(ctx, cmd);
break;
default:
break;
}
}
}
- processRequestCommand(ctx, cmd); 处理请求
- processResponseCommand(ctx, cmd); 处理响应
processRequestCommand
//根据cmd code获取对应的处理pair 如果没有 则使用默认处理器 defaultRequestProcessor
final Pair<NettyRequestProcessor, ExecutorService> matched = this.processorTable.get(cmd.getCode());
final Pair<NettyRequestProcessor, ExecutorService> pair = null == matched ? this.defaultRequestProcessor : matched;
//处理请求
final RemotingCommand response = pair.getObject1().processRequest(ctx, cmd);
1.根据cmd code获取对应的处理pair 如果没有 则使用默认处理器 defaultRequestProcessor
2.处理请求
defaultRequestProcessor 默认处理器的处理逻辑
public RemotingCommand processRequest(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
if (log.isDebugEnabled()) {
log.debug("receive request, {} {} {}",
request.getCode(),
RemotingHelper.parseChannelRemoteAddr(ctx.channel()),
request);
}
switch (request.getCode()) {
//创建或跟新kv配置
case RequestCode.PUT_KV_CONFIG:
return this.putKVConfig(ctx, request);
//获取kv配置
case RequestCode.GET_KV_CONFIG:
return this.getKVConfig(ctx, request);
//删除kv配置
case RequestCode.DELETE_KV_CONFIG:
return this.deleteKVConfig(ctx, request);
//注册broker
case RequestCode.REGISTER_BROKER:
Version brokerVersion = MQVersion.value2Version(request.getVersion());
if (brokerVersion.ordinal() >= MQVersion.Version.V3_0_11.ordinal()) {
return this.registerBrokerWithFilterServer(ctx, request);
} else {
//兼容低版本
return this.registerBroker(ctx, request);
}
//注销broker
case RequestCode.UNREGISTER_BROKER:
return this.unregisterBroker(ctx, request);
//获取指定TOPIC的路由信息
case RequestCode.GET_ROUTEINTO_BY_TOPIC:
return this.getRouteInfoByTopic(ctx, request);
//获取broker集群信息
case RequestCode.GET_BROKER_CLUSTER_INFO:
return this.getBrokerClusterInfo(ctx, request);
//删除一个Broker的写权限
case RequestCode.WIPE_WRITE_PERM_OF_BROKER:
return this.wipeWritePermOfBroker(ctx, request);
//获取所有主题信息
case RequestCode.GET_ALL_TOPIC_LIST_FROM_NAMESERVER:
return getAllTopicListFromNameserver(ctx, request);
//删除主题信息
case RequestCode.DELETE_TOPIC_IN_NAMESRV:
return deleteTopicInNamesrv(ctx, request);
//获取kv配置
case RequestCode.GET_KVLIST_BY_NAMESPACE:
return this.getKVListByNamespace(ctx, request);
//获取指定集群所有的主题
case RequestCode.GET_TOPICS_BY_CLUSTER:
return this.getTopicsByCluster(ctx, request);
//系统会将集群名称、broker名称作为默认topic创建。现在获取这类topic
case RequestCode.GET_SYSTEM_TOPIC_LIST_FROM_NS:
return this.getSystemTopicListFromNs(ctx, request);
case RequestCode.GET_UNIT_TOPIC_LIST:
return this.getUnitTopicList(ctx, request);
case RequestCode.GET_HAS_UNIT_SUB_TOPIC_LIST:
return this.getHasUnitSubTopicList(ctx, request);
case RequestCode.GET_HAS_UNIT_SUB_UNUNIT_TOPIC_LIST:
return this.getHasUnitSubUnUnitTopicList(ctx, request);
//更新properties请求
case RequestCode.UPDATE_NAMESRV_CONFIG:
return this.updateConfig(ctx, request);
//获取properties内容
case RequestCode.GET_NAMESRV_CONFIG:
return this.getConfig(ctx, request);
default:
break;
}
return null;
}
重点看一下REGISTER_BROKER 注册broker的逻辑 registerBrokerWithFilterServer
//注册broker
RegisterBrokerResult result = this.namesrvController.getRouteInfoManager().registerBroker(
requestHeader.getClusterName(),//集群名
requestHeader.getBrokerAddr(),//broker地址
requestHeader.getBrokerName(),//broker名字
requestHeader.getBrokerId(),//brokerId
requestHeader.getHaServerAddr(),//集群地址
registerBrokerBody.getTopicConfigSerializeWrapper(),
registerBrokerBody.getFilterServerList(),
ctx.channel());
可以看到内部是调用RouteInfoManager来注册broker信息，注册的主要信息参数有
- 集群名
- broker地址
- broker名字
- brokerId 0表示主节点
- 集群地址
RouteInfoManager#registerBroker
public RegisterBrokerResult registerBroker(
final String clusterName,//集群名
final String brokerAddr,//broker地址
final String brokerName,//broker名字
final long brokerId,//brokerId
final String haServerAddr,//集群地址
final TopicConfigSerializeWrapper topicConfigWrapper,//主题配置信息
final List<String> filterServerList,//过滤服务器
final Channel channel) {
RegisterBrokerResult result = new RegisterBrokerResult();
try {
try {
//加写锁（读写锁 适合读多写少场景 读读不互斥）
this.lock.writeLock().lockInterruptibly();
//获取集群名为clusterName下的brokerNames
Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
if (null == brokerNames) {
//不存在就创建一个
brokerNames = new HashSet<String>();
this.clusterAddrTable.put(clusterName, brokerNames);
}
//添加当前brokerName
brokerNames.add(brokerName);
boolean registerFirst = false;
//获取brokerName对应的broker数据 brokerData内有主从broker地址信息 brokerId为0的表示主节点
BrokerData brokerData = this.brokerAddrTable.get(brokerName);
if (null == brokerData) {
//第一次注册 添加brokerData信息
registerFirst = true;
brokerData = new BrokerData(clusterName, brokerName, new HashMap<Long, String>());
this.brokerAddrTable.put(brokerName, brokerData);
}
//跟新broker信息
String oldAddr = brokerData.getBrokerAddrs().put(brokerId, brokerAddr);
registerFirst = registerFirst || (null == oldAddr);
if (null != topicConfigWrapper
&& MixAll.MASTER_ID == brokerId) {
//判断borker主题配置信息有无改变 或者是否是第一次注册
if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
|| registerFirst) {
ConcurrentMap<String, TopicConfig> tcTable =
topicConfigWrapper.getTopicConfigTable();
if (tcTable != null) {
//跟新topic配置信息
for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
this.createAndUpdateQueueData(brokerName, entry.getValue());
}
}
}
}
//跟新broker的心跳存活信息
BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
new BrokerLiveInfo(
System.currentTimeMillis(),
topicConfigWrapper.getDataVersion(),
channel,
haServerAddr));
if (null == prevBrokerLiveInfo) {
log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
}
//跟新过滤服务器信息
if (filterServerList != null) {
if (filterServerList.isEmpty()) {
this.filterServerTable.remove(brokerAddr);
} else {
this.filterServerTable.put(brokerAddr, filterServerList);
}
}
//如果不是主节点 则找出主节点并将返回结果设置HaServerAddr地址和MasterAddr地址
if (MixAll.MASTER_ID != brokerId) {
String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
if (masterAddr != null) {
BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
if (brokerLiveInfo != null) {
result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
result.setMasterAddr(masterAddr);
}
}
}
} finally {
this.lock.writeLock().unlock();
}
} catch (Exception e) {
log.error("registerBroker Exception", e);
}
return result;
}
RouteInfoManager 路由信息管理器
关注一下路由信息管理里面的几个核心对象属性
1.HashMap<String/* topic */, List
2.HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
3.HashMap<String/* clusterName /, Set<String/ brokerName */>> clusterAddrTable;
4.HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
5.HashMap<String/* brokerAddr /, List
topicQueueTable 主题队列信息
topicQueueTable保存了主题在哪些 broker上注册，每个broker对应的读写队列数量以及读写权限
key为主题topic，value为List
QueueData属性如下
//broker名字
private String brokerName;
//一般来说读写队列数量一致，如果不一致就会出现很多问题。
//读队列数量
private int readQueueNums;
//写队列数量
private int writeQueueNums;
//6：同时支持读写 4：禁写 2：禁读
private int perm;
private int topicSynFlag;
1.brokerName broker名字
2.readQueueNums 读队列数量
3.writeQueueNums 写队列数量
4.perm 同时支持读写 4：禁写 2：禁读
5.topicSynFlag 尚不明确
brokerAddrTable broker地址信息
key为brokerName，value为BrokerData
brokerAddrTable保存了brokerName对应的主从节点地址信息
BrokerData属性如下：
private String cluster;
private String brokerName;
private HashMap<Long/* brokerId */, String/* broker address */> brokerAddrs;
1.cluster 所属集群
- brokerName broker名字
- brokerAddrs key为brokerId，value为broker的地址，brokerId=0表示主节点
clusterAddrTable 集群地址信息
clusterAddrTable保存了集群对应的brokerName集合
brokerLiveTable broker存活信息
brokerLiveTable保存了 broker的存活信息
key为brokerAddr broker地址 value为BrokerLiveInfo
BrokerLiveInfo属性如下：
private long lastUpdateTimestamp;
private DataVersion dataVersion;
private Channel channel;
private String haServerAddr;
1.lastUpdateTimestamp 最后一次跟新时间戳
2.dataVersion 数据版本
3.channel 通信通道
4.haServerAddr 集群地址
namesrv会定期的检测brokerLiveInfo，将2分钟没有跟新brokerLiveInfo的broker地址剔除
//扫描不活跃的broker
public void scanNotActiveBroker() {
Iterator<Entry<String, BrokerLiveInfo>> it = this.brokerLiveTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, BrokerLiveInfo> next = it.next();
long last = next.getValue().getLastUpdateTimestamp();
//关闭2分钟未发送心跳信息的broker
if ((last + BROKER_CHANNEL_EXPIRED_TIME) < System.currentTimeMillis()) {
//关闭netty channel
RemotingUtil.closeChannel(next.getValue().getChannel());
//剔除broker信息
it.remove();
log.warn("The broker channel expired, {} {}ms", next.getKey(), BROKER_CHANNEL_EXPIRED_TIME);
this.onChannelDestroy(next.getKey(), next.getValue().getChannel());
}
}
}
filterServerTable 过滤服务器地址
filterServerTable保存了broker地址对应的过滤服务器地址
key为brokerAddr broker地址，value为List
