RocketMQ(4.8.0)——Broker读写分离机制

Broker读写分离机制

　　在 RocketMQ 中，有2处地方使用到 "读写分离" 机制。

　　Broker Master-Slave 读写分离：写操作到 Master Broker，从 Slave Broker 读取消息。Broker 配置为 slaveReadEnable=True（默认False），消息占用内存百度分配置为 accessMessageInMemoryMaxRatio=40（默认）。

　　Broker Direct Memory-Page Cache 读写：写消息到 Direct Memory（直接内存，简称 DM），从操作系统的 Page Cache 中读取消息。Master Broker 配置读写分离开关为 tranientStorePoolEnable=True（默认为False），写入 DM 存储数量，配置 trainsientStorePoolSize 至少大于0（默认为5，建议不修改），刷盘类型配置为 flushDiskType=FlushDiskType.ASYNC_FLUSH，即异步输盘。

　　首先我们来讲 Master-Slave 读写分离机制。通常，都是 Master 提供读写处理，如果 Master 负载较高，就从 Slave 读取，整个过程如下：

　　该机制的实现分为以下两个步骤。

　　第一步：Broker 在处理 Pull 消息时，计算下次是否从 Slave 拉取消息，是通过 org.apache.rocketmq.store.DefaultMessageStore.getMessage() 方法实现的，代码路径：D:\rocketmq-master\store\src\main\java\org\apache\rocketmq\store\DefaultMessageStore.java，代码如下：

 1                 SelectMappedBufferResult bufferConsumeQueue = consumeQueue.getIndexBuffer(offset);
 2                 if (bufferConsumeQueue != null) {
 3                     try {
 4                         status = GetMessageStatus.NO_MATCHED_MESSAGE;
 5 
 6                         long nextPhyFileStartOffset = Long.MIN_VALUE;
 7                         long maxPhyOffsetPulling = 0;   #表示拉取的最大消息位点。
 8 
 9                         int i = 0;
10                         final int maxFilterMessageCount = Math.max(16000, maxMsgNums * ConsumeQueue.CQ_STORE_UNIT_SIZE);
11                         final boolean diskFallRecorded = this.messageStoreConfig.isDiskFallRecorded();
12                         ConsumeQueueExt.CqExtUnit cqExtUnit = new ConsumeQueueExt.CqExtUnit();
13                         for (; i < bufferConsumeQueue.getSize() && i < maxFilterMessageCount; i += ConsumeQueue.CQ_STORE_UNIT_SIZE) {
14                             long offsetPy = bufferConsumeQueue.getByteBuffer().getLong();
15                             int sizePy = bufferConsumeQueue.getByteBuffer().getInt();
16                             long tagsCode = bufferConsumeQueue.getByteBuffer().getLong();
17 
18                             maxPhyOffsetPulling = offsetPy;
19 
20                             if (nextPhyFileStartOffset != Long.MIN_VALUE) {
21                                 if (offsetPy < nextPhyFileStartOffset)
22                                     continue;
23                             }
24 
25                             boolean isInDisk = checkInDiskByCommitOffset(offsetPy, maxOffsetPy);  #表示当前 Master Broker 存储的所有消息的最大物理位点
26 
27                             if (this.isTheBatchFull(sizePy, maxMsgNums, getResult.getBufferTotalSize(), getResult.getMessageCount(),
28                                 isInDisk)) {
29                                 break;
30                             }
31 
32                             boolean extRet = false, isTagsCodeLegal = true;
33                             if (consumeQueue.isExtAddr(tagsCode)) {
34                                 extRet = consumeQueue.getExt(tagsCode, cqExtUnit);
35                                 if (extRet) {
36                                     tagsCode = cqExtUnit.getTagsCode();
37                                 } else {
38                                     // can't find ext content.Client will filter messages by tag also.
39                                     log.error("[BUG] can't find consume queue extend file content!addr={}, offsetPy={}, sizePy={}, topic={}, group={}",
40                                         tagsCode, offsetPy, sizePy, topic, group);
41                                     isTagsCodeLegal = false;
42                                 }
43                             }
44 
45                             if (messageFilter != null
46                                 && !messageFilter.isMatchedByConsumeQueue(isTagsCodeLegal ? tagsCode : null, extRet ? cqExtUnit : null)) {
47                                 if (getResult.getBufferTotalSize() == 0) {
48                                     status = GetMessageStatus.NO_MATCHED_MESSAGE;
49                                 }
50 
51                                 continue;
52                             }
53 
54                             SelectMappedBufferResult selectResult = this.commitLog.getMessage(offsetPy, sizePy);
55                             if (null == selectResult) {
56                                 if (getResult.getBufferTotalSize() == 0) {
57                                     status = GetMessageStatus.MESSAGE_WAS_REMOVING;
58                                 }
59 
60                                 nextPhyFileStartOffset = this.commitLog.rollNextFile(offsetPy);
61                                 continue;
62                             }
63 
64                             if (messageFilter != null
65                                 && !messageFilter.isMatchedByCommitLog(selectResult.getByteBuffer().slice(), null)) {
66                                 if (getResult.getBufferTotalSize() == 0) {
67                                     status = GetMessageStatus.NO_MATCHED_MESSAGE;
68                                 }
69                                 // release...
70                                 selectResult.release();
71                                 continue;
72                             }
73 
74                             this.storeStatsService.getGetMessageTransferedMsgCount().incrementAndGet();
75                             getResult.addMessage(selectResult);
76                             status = GetMessageStatus.FOUND;
77                             nextPhyFileStartOffset = Long.MIN_VALUE;
78                         }
79 
80                         if (diskFallRecorded) {
81                             long fallBehind = maxOffsetPy - maxPhyOffsetPulling;
82                             brokerStatsManager.recordDiskFallBehindSize(group, topic, queueId, fallBehind);
83                         }
84 
85                         nextBeginOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE);
86 
87                         long diff = maxOffsetPy - maxPhyOffsetPulling;  #diff：是 maxOffsetPy 和 maxPhyOffsetPulling 两者的差值，表示还有多少消息没有拉取
88                         long memory = (long) (StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE #StoreUtil.TOTAL_PHYSICAL_MEMORY_SIZE：表示当前 Master Broker 全部的物理内存大小。

　　　　　　　#memory：Broker 认为可使用最大内存，该值可以通过 accessMessageInMemoryMaxRatio 配置项决定，默认 accessMessageInMemoryMaxRatio=40，如果物理内存为 100MB，那么memory=40MB。
89                             * (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0)); 
                           #设置下次从 Master 或 Slave 拉取消息
90                         getResult.setSuggestPullingFromSlave(diff > memory);#表示没有拉取的消息比分配的内存大，如果diff>memory的值为True，则说明此事Master Broker内存繁忙，该从slave拉
91                     } finally {
92 
93                         bufferConsumeQueue.release();
94                     }
95                 }

　　第二步：通知客户端下次从哪个 Broker 拉取消息。在消费者 Pull 消息返回结果时，根据第一步设置的 suggestPullingFromSlave 值返回给消费者，该过程通过 D:\rocketmq-master\broker\src\main\java\org\apache\rocketmq\broker\processor\PullMessageProcessor.java 中 processRequest()方法实现，具体代码如下：

  1 private RemotingCommand processRequest(final Channel channel, RemotingCommand request, boolean brokerAllowSuspend)
  2         throws RemotingCommandException {
  3         RemotingCommand response = RemotingCommand.createResponseCommand(PullMessageResponseHeader.class);
  4         final PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) response.readCustomHeader();
  5         final PullMessageRequestHeader requestHeader =
  6             (PullMessageRequestHeader) request.decodeCommandCustomHeader(PullMessageRequestHeader.class);
  7 
  8         response.setOpaque(request.getOpaque());
  9 
 10         log.debug("receive PullMessage request command, {}", request);
 11 
 12         if (!PermName.isReadable(this.brokerController.getBrokerConfig().getBrokerPermission())) {
 13             response.setCode(ResponseCode.NO_PERMISSION);
 14             response.setRemark(String.format("the broker[%s] pulling message is forbidden", this.brokerController.getBrokerConfig().getBrokerIP1()));
 15             return response;
 16         }
 17 
 18         SubscriptionGroupConfig subscriptionGroupConfig =
 19             this.brokerController.getSubscriptionGroupManager().findSubscriptionGroupConfig(requestHeader.getConsumerGroup());
 20         if (null == subscriptionGroupConfig) {
 21             response.setCode(ResponseCode.SUBSCRIPTION_GROUP_NOT_EXIST);
 22             response.setRemark(String.format("subscription group [%s] does not exist, %s", requestHeader.getConsumerGroup(), FAQUrl.suggestTodo(FAQUrl.SUBSCRIPTION_GROUP_NOT_EXIST)));
 23             return response;
 24         }
 25 
 26         if (!subscriptionGroupConfig.isConsumeEnable()) {
 27             response.setCode(ResponseCode.NO_PERMISSION);
 28             response.setRemark("subscription group no permission, " + requestHeader.getConsumerGroup());
 29             return response;
 30         }
 31 
 32         final boolean hasSuspendFlag = PullSysFlag.hasSuspendFlag(requestHeader.getSysFlag());
 33         final boolean hasCommitOffsetFlag = PullSysFlag.hasCommitOffsetFlag(requestHeader.getSysFlag());
 34         final boolean hasSubscriptionFlag = PullSysFlag.hasSubscriptionFlag(requestHeader.getSysFlag());
 35 
 36         final long suspendTimeoutMillisLong = hasSuspendFlag ? requestHeader.getSuspendTimeoutMillis() : 0;
 37 
 38         TopicConfig topicConfig = this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic());
 39         if (null == topicConfig) {
 40             log.error("the topic {} not exist, consumer: {}", requestHeader.getTopic(), RemotingHelper.parseChannelRemoteAddr(channel));
 41             response.setCode(ResponseCode.TOPIC_NOT_EXIST);
 42             response.setRemark(String.format("topic[%s] not exist, apply first please! %s", requestHeader.getTopic(), FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL)));
 43             return response;
 44         }
 45 
 46         if (!PermName.isReadable(topicConfig.getPerm())) {
 47             response.setCode(ResponseCode.NO_PERMISSION);
 48             response.setRemark("the topic[" + requestHeader.getTopic() + "] pulling message is forbidden");
 49             return response;
 50         }
 51 
 52         if (requestHeader.getQueueId() < 0 || requestHeader.getQueueId() >= topicConfig.getReadQueueNums()) {
 53             String errorInfo = String.format("queueId[%d] is illegal, topic:[%s] topicConfig.readQueueNums:[%d] consumer:[%s]",
 54                 requestHeader.getQueueId(), requestHeader.getTopic(), topicConfig.getReadQueueNums(), channel.remoteAddress());
 55             log.warn(errorInfo);
 56             response.setCode(ResponseCode.SYSTEM_ERROR);
 57             response.setRemark(errorInfo);
 58             return response;
 59         }
 60 
 61         SubscriptionData subscriptionData = null;
 62         ConsumerFilterData consumerFilterData = null;
 63         if (hasSubscriptionFlag) {
 64             try {
 65                 subscriptionData = FilterAPI.build(
 66                     requestHeader.getTopic(), requestHeader.getSubscription(), requestHeader.getExpressionType()
 67                 );
 68                 if (!ExpressionType.isTagType(subscriptionData.getExpressionType())) {
 69                     consumerFilterData = ConsumerFilterManager.build(
 70                         requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getSubscription(),
 71                         requestHeader.getExpressionType(), requestHeader.getSubVersion()
 72                     );
 73                     assert consumerFilterData != null;
 74                 }
 75             } catch (Exception e) {
 76                 log.warn("Parse the consumer's subscription[{}] failed, group: {}", requestHeader.getSubscription(),
 77                     requestHeader.getConsumerGroup());
 78                 response.setCode(ResponseCode.SUBSCRIPTION_PARSE_FAILED);
 79                 response.setRemark("parse the consumer's subscription failed");
 80                 return response;
 81             }
 82         } else {
 83             ConsumerGroupInfo consumerGroupInfo =
 84                 this.brokerController.getConsumerManager().getConsumerGroupInfo(requestHeader.getConsumerGroup());
 85             if (null == consumerGroupInfo) {
 86                 log.warn("the consumer's group info not exist, group: {}", requestHeader.getConsumerGroup());
 87                 response.setCode(ResponseCode.SUBSCRIPTION_NOT_EXIST);
 88                 response.setRemark("the consumer's group info not exist" + FAQUrl.suggestTodo(FAQUrl.SAME_GROUP_DIFFERENT_TOPIC));
 89                 return response;
 90             }
 91 
 92             if (!subscriptionGroupConfig.isConsumeBroadcastEnable()
 93                 && consumerGroupInfo.getMessageModel() == MessageModel.BROADCASTING) {
 94                 response.setCode(ResponseCode.NO_PERMISSION);
 95                 response.setRemark("the consumer group[" + requestHeader.getConsumerGroup() + "] can not consume by broadcast way");
 96                 return response;
 97             }
 98 
 99             subscriptionData = consumerGroupInfo.findSubscriptionData(requestHeader.getTopic());
100             if (null == subscriptionData) {
101                 log.warn("the consumer's subscription not exist, group: {}, topic:{}", requestHeader.getConsumerGroup(), requestHeader.getTopic());
102                 response.setCode(ResponseCode.SUBSCRIPTION_NOT_EXIST);
103                 response.setRemark("the consumer's subscription not exist" + FAQUrl.suggestTodo(FAQUrl.SAME_GROUP_DIFFERENT_TOPIC));
104                 return response;
105             }
106 
107             if (subscriptionData.getSubVersion() < requestHeader.getSubVersion()) {
108                 log.warn("The broker's subscription is not latest, group: {} {}", requestHeader.getConsumerGroup(),
109                     subscriptionData.getSubString());
110                 response.setCode(ResponseCode.SUBSCRIPTION_NOT_LATEST);
111                 response.setRemark("the consumer's subscription not latest");
112                 return response;
113             }
114             if (!ExpressionType.isTagType(subscriptionData.getExpressionType())) {
115                 consumerFilterData = this.brokerController.getConsumerFilterManager().get(requestHeader.getTopic(),
116                     requestHeader.getConsumerGroup());
117                 if (consumerFilterData == null) {
118                     response.setCode(ResponseCode.FILTER_DATA_NOT_EXIST);
119                     response.setRemark("The broker's consumer filter data is not exist!Your expression may be wrong!");
120                     return response;
121                 }
122                 if (consumerFilterData.getClientVersion() < requestHeader.getSubVersion()) {
123                     log.warn("The broker's consumer filter data is not latest, group: {}, topic: {}, serverV: {}, clientV: {}",
124                         requestHeader.getConsumerGroup(), requestHeader.getTopic(), consumerFilterData.getClientVersion(), requestHeader.getSubVersion());
125                     response.setCode(ResponseCode.FILTER_DATA_NOT_LATEST);
126                     response.setRemark("the consumer's consumer filter data not latest");
127                     return response;
128                 }
129             }
130         }
131 
132         if (!ExpressionType.isTagType(subscriptionData.getExpressionType())
133             && !this.brokerController.getBrokerConfig().isEnablePropertyFilter()) {
134             response.setCode(ResponseCode.SYSTEM_ERROR);
135             response.setRemark("The broker does not support consumer to filter message by " + subscriptionData.getExpressionType());
136             return response;
137         }
138 
139         MessageFilter messageFilter;
140         if (this.brokerController.getBrokerConfig().isFilterSupportRetry()) {
141             messageFilter = new ExpressionForRetryMessageFilter(subscriptionData, consumerFilterData,
142                 this.brokerController.getConsumerFilterManager());
143         } else {
144             messageFilter = new ExpressionMessageFilter(subscriptionData, consumerFilterData,
145                 this.brokerController.getConsumerFilterManager());
146         }
147 
148         final GetMessageResult getMessageResult =
149             this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
150                 requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter);
151         if (getMessageResult != null) {
152             response.setRemark(getMessageResult.getStatus().name());
153             responseHeader.setNextBeginOffset(getMessageResult.getNextBeginOffset());
154             responseHeader.setMinOffset(getMessageResult.getMinOffset());
155             responseHeader.setMaxOffset(getMessageResult.getMaxOffset());
156 
157             if (getMessageResult.isSuggestPullingFromSlave()) {
158                 responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly());
159             } else {
160                 responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
161             }
162 
163             switch (this.brokerController.getMessageStoreConfig().getBrokerRole()) {
164                 case ASYNC_MASTER:
165                 case SYNC_MASTER:
166                     break;
167                 case SLAVE:
168                     if (!this.brokerController.getBrokerConfig().isSlaveReadEnable()) {
169                         response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
170                         responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
171                     }
172                     break;
173             }
174 　　　　　　　# slave 读取开关配置
175             if (this.brokerController.getBrokerConfig().isSlaveReadEnable()) {
176                 // consume too slow ,redirect to another machine
177                 if (getMessageResult.isSuggestPullingFromSlave()) {
178                     responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly());
179                 }
180                 // consume ok
181                 else {
182                     responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getBrokerId());
183                 }
184             } else {
185                 responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
186             }
187 
188             switch (getMessageResult.getStatus()) {
189                 case FOUND:
190                     response.setCode(ResponseCode.SUCCESS);
191                     break;
192                 case MESSAGE_WAS_REMOVING:
193                     response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
194                     break;
195                 case NO_MATCHED_LOGIC_QUEUE:
196                 case NO_MESSAGE_IN_QUEUE:
197                     if (0 != requestHeader.getQueueOffset()) {
198                         response.setCode(ResponseCode.PULL_OFFSET_MOVED);
199 
200                         // XXX: warn and notify me
201                         log.info("the broker store no queue data, fix the request offset {} to {}, Topic: {} QueueId: {} Consumer Group: {}",
202                             requestHeader.getQueueOffset(),
203                             getMessageResult.getNextBeginOffset(),
204                             requestHeader.getTopic(),
205                             requestHeader.getQueueId(),
206                             requestHeader.getConsumerGroup()
207                         );
208                     } else {
209                         response.setCode(ResponseCode.PULL_NOT_FOUND);
210                     }
211                     break;
212                 case NO_MATCHED_MESSAGE:
213                     response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
214                     break;
215                 case OFFSET_FOUND_NULL:
216                     response.setCode(ResponseCode.PULL_NOT_FOUND);
217                     break;
218                 case OFFSET_OVERFLOW_BADLY:
219                     response.setCode(ResponseCode.PULL_OFFSET_MOVED);
220                     // XXX: warn and notify me
221                     log.info("the request offset: {} over flow badly, broker max offset: {}, consumer: {}",
222                         requestHeader.getQueueOffset(), getMessageResult.getMaxOffset(), channel.remoteAddress());
223                     break;
224                 case OFFSET_OVERFLOW_ONE:
225                     response.setCode(ResponseCode.PULL_NOT_FOUND);
226                     break;
227                 case OFFSET_TOO_SMALL:
228                     response.setCode(ResponseCode.PULL_OFFSET_MOVED);
229                     log.info("the request offset too small. group={}, topic={}, requestOffset={}, brokerMinOffset={}, clientIp={}",
230                         requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueOffset(),
231                         getMessageResult.getMinOffset(), channel.remoteAddress());
232                     break;
233                 default:
234                     assert false;
235                     break;
236             }
237 
238             if (this.hasConsumeMessageHook()) {
239                 ConsumeMessageContext context = new ConsumeMessageContext();
240                 context.setConsumerGroup(requestHeader.getConsumerGroup());
241                 context.setTopic(requestHeader.getTopic());
242                 context.setQueueId(requestHeader.getQueueId());
243 
244                 String owner = request.getExtFields().get(BrokerStatsManager.COMMERCIAL_OWNER);
245 
246                 switch (response.getCode()) {
247                     case ResponseCode.SUCCESS:
248                         int commercialBaseCount = brokerController.getBrokerConfig().getCommercialBaseCount();
249                         int incValue = getMessageResult.getMsgCount4Commercial() * commercialBaseCount;
250 
251                         context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_SUCCESS);
252                         context.setCommercialRcvTimes(incValue);
253                         context.setCommercialRcvSize(getMessageResult.getBufferTotalSize());
254                         context.setCommercialOwner(owner);
255 
256                         break;
257                     case ResponseCode.PULL_NOT_FOUND:
258                         if (!brokerAllowSuspend) {
259 
260                             context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_EPOLLS);
261                             context.setCommercialRcvTimes(1);
262                             context.setCommercialOwner(owner);
263 
264                         }
265                         break;
266                     case ResponseCode.PULL_RETRY_IMMEDIATELY:
267                     case ResponseCode.PULL_OFFSET_MOVED:
268                         context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_EPOLLS);
269                         context.setCommercialRcvTimes(1);
270                         context.setCommercialOwner(owner);
271                         break;
272                     default:
273                         assert false;
274                         break;
275                 }
276 
277                 this.executeConsumeMessageHookBefore(context);
278             }
279 
280             switch (response.getCode()) {
281                 case ResponseCode.SUCCESS:
282 
283                     this.brokerController.getBrokerStatsManager().incGroupGetNums(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
284                         getMessageResult.getMessageCount());
285 
286                     this.brokerController.getBrokerStatsManager().incGroupGetSize(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
287                         getMessageResult.getBufferTotalSize());
288 
289                     this.brokerController.getBrokerStatsManager().incBrokerGetNums(getMessageResult.getMessageCount());
290                     if (this.brokerController.getBrokerConfig().isTransferMsgByHeap()) {
291                         final long beginTimeMills = this.brokerController.getMessageStore().now();
292                         final byte[] r = this.readGetMessageResult(getMessageResult, requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId());
293                         this.brokerController.getBrokerStatsManager().incGroupGetLatency(requestHeader.getConsumerGroup(),
294                             requestHeader.getTopic(), requestHeader.getQueueId(),
295                             (int) (this.brokerController.getMessageStore().now() - beginTimeMills));
296                         response.setBody(r);
297                     } else {
298                         try {
299                             FileRegion fileRegion =
300                                 new ManyMessageTransfer(response.encodeHeader(getMessageResult.getBufferTotalSize()), getMessageResult);
301                             channel.writeAndFlush(fileRegion).addListener(new ChannelFutureListener() {
302                                 @Override
303                                 public void operationComplete(ChannelFuture future) throws Exception {
304                                     getMessageResult.release();
305                                     if (!future.isSuccess()) {
306                                         log.error("transfer many message by pagecache failed, {}", channel.remoteAddress(), future.cause());
307                                     }
308                                 }
309                             });
310                         } catch (Throwable e) {
311                             log.error("transfer many message by pagecache exception", e);
312                             getMessageResult.release();
313                         }
314 
315                         response = null;
316                     }
317                     break;
318                 case ResponseCode.PULL_NOT_FOUND:
319 
320                     if (brokerAllowSuspend && hasSuspendFlag) {
321                         long pollingTimeMills = suspendTimeoutMillisLong;
322                         if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
323                             pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
324                         }
325 
326                         String topic = requestHeader.getTopic();
327                         long offset = requestHeader.getQueueOffset();
328                         int queueId = requestHeader.getQueueId();
329                         PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills,
330                             this.brokerController.getMessageStore().now(), offset, subscriptionData, messageFilter);
331                         this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
332                         response = null;
333                         break;
334                     }
335 
336                 case ResponseCode.PULL_RETRY_IMMEDIATELY:
337                     break;
338                 case ResponseCode.PULL_OFFSET_MOVED:
339                     if (this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE
340                         || this.brokerController.getMessageStoreConfig().isOffsetCheckInSlave()) {
341                         MessageQueue mq = new MessageQueue();
342                         mq.setTopic(requestHeader.getTopic());
343                         mq.setQueueId(requestHeader.getQueueId());
344                         mq.setBrokerName(this.brokerController.getBrokerConfig().getBrokerName());
345 
346                         OffsetMovedEvent event = new OffsetMovedEvent();
347                         event.setConsumerGroup(requestHeader.getConsumerGroup());
348                         event.setMessageQueue(mq);
349                         event.setOffsetRequest(requestHeader.getQueueOffset());
350                         event.setOffsetNew(getMessageResult.getNextBeginOffset());
351                         this.generateOffsetMovedEvent(event);
352                         log.warn(
353                             "PULL_OFFSET_MOVED:correction offset. topic={}, groupId={}, requestOffset={}, newOffset={}, suggestBrokerId={}",
354                             requestHeader.getTopic(), requestHeader.getConsumerGroup(), event.getOffsetRequest(), event.getOffsetNew(),
355                             responseHeader.getSuggestWhichBrokerId());
356                     } else {
357                         responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getBrokerId());
358                         response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
359                         log.warn("PULL_OFFSET_MOVED:none correction. topic={}, groupId={}, requestOffset={}, suggestBrokerId={}",
360                             requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getQueueOffset(),
361                             responseHeader.getSuggestWhichBrokerId());
362                     }
363 
364                     break;
365                 default:
366                     assert false;
367             }
368         } else {
369             response.setCode(ResponseCode.SYSTEM_ERROR);
370             response.setRemark("store getMessage return null");
371         }
372 
373         boolean storeOffsetEnable = brokerAllowSuspend;
374         storeOffsetEnable = storeOffsetEnable && hasCommitOffsetFlag;
375         storeOffsetEnable = storeOffsetEnable
376             && this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE;
377         if (storeOffsetEnable) {
378             this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(channel),
379                 requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset());
380         }
381         return response;
382     }

　　通过查看以上代码第168行，我们知道要想从 Slave 读取消息，需要设置 slaveReadEnable=True，此时会根据第一步返回的 suggestPullingFromSlave 值告诉客户端下次可以从哪个 Broker 拉取消息。suggestPullingFromSlave=1 表示从 Slave 拉取，suggestPullingFromSlave=0 表示从 Master 拉取。

　　在了解了 Master-Slave 读写分离机制后，接着讲 Direct Memory-Page Cache 的读写分离机制：

　　以上逻辑通过 D:\rocketmq-master\store\src\main\java\org\apache\rocketmq\store\MappedFile.java 中 appendMessagesInner()，具体代码如下：

 1     public AppendMessageResult appendMessagesInner(final MessageExt messageExt, final AppendMessageCallback cb) {
 2         assert messageExt != null;
 3         assert cb != null;
 4 
 5         int currentPos = this.wrotePosition.get();
 6 
 7         if (currentPos < this.fileSize) {
 8             ByteBuffer byteBuffer = writeBuffer != null ? writeBuffer.slice() : this.mappedByteBuffer.slice();
 9             byteBuffer.position(currentPos);
10             AppendMessageResult result;
11             if (messageExt instanceof MessageExtBrokerInner) {
12                 result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBrokerInner) messageExt);
13             } else if (messageExt instanceof MessageExtBatch) {
14                 result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBatch) messageExt);
15             } else {
16                 return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR);
17             }
18             this.wrotePosition.addAndGet(result.getWroteBytes());
19             this.storeTimestamp = result.getStoreTimestamp();
20             return result;
21         }
22         log.error("MappedFile.appendMessage return null, wrotePosition: {} fileSize: {}", currentPos, this.fileSize);
23         return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR);
24     }

　　这段代码中，writeBuffer 表示从 DM 中申请的缓存；mappendByteBuffer 表示从 Page Cache 中申请的缓存。如果 Broker 设置 transientStorePoolEnable=true，并且异步刷盘，则存储层src\main\java\org\apache\rocketmq\store\DefaultMessageStore.java 在初始化时会调用 TransientStorePool.init() 方法（按照配置的 Buffer 个数）初始化 writeBuffer。代码如下：

 1     public DefaultMessageStore(final MessageStoreConfig messageStoreConfig, final BrokerStatsManager brokerStatsManager,
 2         final MessageArrivingListener messageArrivingListener, final BrokerConfig brokerConfig) throws IOException {
 3         this.messageArrivingListener = messageArrivingListener;
 4         this.brokerConfig = brokerConfig;
 5         this.messageStoreConfig = messageStoreConfig;
 6         this.brokerStatsManager = brokerStatsManager;
 7         this.allocateMappedFileService = new AllocateMappedFileService(this);
 8         if (messageStoreConfig.isEnableDLegerCommitLog()) {
 9             this.commitLog = new DLedgerCommitLog(this);
10         } else {
11             this.commitLog = new CommitLog(this);
12         }
13         this.consumeQueueTable = new ConcurrentHashMap<>(32);
14 
15         this.flushConsumeQueueService = new FlushConsumeQueueService();
16         this.cleanCommitLogService = new CleanCommitLogService();
17         this.cleanConsumeQueueService = new CleanConsumeQueueService();
18         this.storeStatsService = new StoreStatsService();
19         this.indexService = new IndexService(this);
20         if (!messageStoreConfig.isEnableDLegerCommitLog()) {
21             this.haService = new HAService(this);
22         } else {
23             this.haService = null;
24         }
25         this.reputMessageService = new ReputMessageService();
26 
27         this.scheduleMessageService = new ScheduleMessageService(this);
28 
29         this.transientStorePool = new TransientStorePool(messageStoreConfig);
30 
31         if (messageStoreConfig.isTransientStorePoolEnable()) {
32             this.transientStorePool.init();
33         }
34 
35         this.allocateMappedFileService.start();
36 
37         this.indexService.start();
38 
39         this.dispatcherList = new LinkedList<>();
40         this.dispatcherList.addLast(new CommitLogDispatcherBuildConsumeQueue());
41         this.dispatcherList.addLast(new CommitLogDispatcherBuildIndex());
42 
43         File file = new File(StorePathConfigHelper.getLockFile(messageStoreConfig.getStorePathRootDir()));
44         MappedFile.ensureDirOK(file.getParent());
45         lockFile = new RandomAccessFile(file, "rw");
46     }

　　初始化 writeBuffer 后，当生产者将消息发送到 Broker时，Broker 将消息写入 writeBuffer，然后被异步转存服务不断地从 DM 中 Commit 到 Page Cache 中。消费者此时从哪儿读取数据呢？消费者拉取消息的实现在 D:\rocketmq-master\store\src\main\java\org\apache\rocketmq\store\MappedFile.java 中 selectMappedBuffer() 方法中，具体代码如下：

1             if (this.hold()) {
2                 ByteBuffer byteBuffer = this.mappedByteBuffer.slice();
3                 byteBuffer.position(pos);
4                 ByteBuffer byteBufferNew = byteBuffer.slice();
5                 byteBufferNew.limit(size);
6                 return new SelectMappedBufferResult(this.fileFromOffset + pos, byteBufferNew, size, this);

　　从代码中可以看到，消费者始终从 mappedByteBuffer（即 Page Cache）读取消息。

　　读写分离能够最大限度地提供吞吐量，同时会增加数据不一致性的风险，建议生产环境谨慎使用。

posted @ 2021-03-01 14:27 左扬阅读(1526) 评论(0) 编辑收藏举报

会员力量，点亮园子希望

刷新页面返回顶部

左扬(你们的胃叫胃，孤的叫胃PLUS)

读书不觉春已深，一寸光阴一寸金。

RocketMQ(4.8.0)——Broker读写分离机制

Broker读写分离机制

公告