【RocketMQ】消息消费流程

概述

消息的消费是一个先从Broke拉到client端，再consume的过程
客户端有一个PullMessageService线程拉取消息，然后把消息放到缓存中（红黑树结构），然后启动 ConsumeMessageService线程消费这些消息，这个过程会使用Consumer启动时注册的Listener消费，

@1：PullMessageService#run

while (!this.isStopped()) {
    try {
        PullRequest pullRequest = this.pullRequestQueue.take();
        this.pullMessage(pullRequest);
    } catch (InterruptedException ignored) {
    } catch (Exception e) {
        log.error("Pull Message Service Run Method exception", e);
    }
}

这个线程不断地从PullRequestQueue中拿到pull的请求，然后执行pullMessage方法
todo：何时加入PullRequestQueue呢？

@2：DefaultMQPushConsumerImpl#pullMessage(PullRequest)
逻辑如下：

1. 如果ProcessQueue被丢弃，不做处理
2. 如果服务被暂停，稍后再拉取executePullRequestLater
3. 做一些流控：

• 缓存的信息超过1000条，稍后再拉
• 缓存的消息大小超过100MB...
• 并发消费时，如果processQueue中第一条消息和最后一条消息的offset超过2000就slow pull；
• 顺序消费时，如果对了被锁定了，重设offset，否则slow pull
  processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()
• 获取订阅消息为空，3s后再拉
  出现这些情况说明消费端的能力跟不上或者内存消耗过大，采用降低pull频率来保证服务

@3：this.pullAPIWrapper.pullKernelImpl方法
向Broker拉取消息，这个方法传入了一个回调函数pullCallback，在Broker中发现消息时被调用

this.pullAPIWrapper.pullKernelImpl(
    pullRequest.getMessageQueue(),
    subExpression,
    subscriptionData.getExpressionType(),
    subscriptionData.getSubVersion(),
    pullRequest.getNextOffset(),
    this.defaultMQPushConsumer.getPullBatchSize(),
    sysFlag,
    commitOffsetValue,
    BROKER_SUSPEND_MAX_TIME_MILLIS,
    CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND,
    CommunicationMode.ASYNC,
    pullCallback
);

---

PullCallback#onSuccess

FOUND:

1. 更新下次拉取的offset（这个pullRequest会重新被放到线程池执行）
2. 统计pull的RT
3. 往processQueue中添加拉取的消息
4. 提交消费任务ConsumeRequest`

boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest

NO_NEW_MSG：重新拉取
NO_MATCHED_MSG: 重新拉取
OFFSET_ILLEGAL: 修改下次拉取的offset后重新拉取

---

消费过程

@1 ConsumeMessageService#submitConsumeRequest

DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
    pullResult.getMsgFoundList(),
    processQueue,
    pullRequest.getMessageQueue(),
    dispatchToConsume);

默认一条一条消费，可在消费者启动时通过setConsumeMessageBatchMaxSize()参数指定

ConsumeRequest consumeRequest = new ConsumeRequest(msgs, processQueue, messageQueue);
this.consumeExecutor.submit(consumeRequest);

其中consumeRequest是一个Runnable对象，这个里面调用了listener来消费
ConsumeRequest@run

public void run() {
//如果ProcessQueue被丢弃就结束本次处理
if (this.processQueue.isDropped()) {
    log.info("the message queue not be able to consume, because it's dropped. group={} {}", ConsumeMessageConcurrentlyService.this.consumerGroup, this.messageQueue);
    return;
}*
 
MessageListenerConcurrently listener = ConsumeMessageConcurrentlyService.this.messageListener;
ConsumeConcurrentlyContext context = new ConsumeConcurrentlyContext(messageQueue);
ConsumeConcurrentlyStatus status = null;
 
ConsumeMessageContext consumeMessageContext = null;
//钩子函数
...
long beginTimestamp = System.currentTimeMillis();
boolean hasException = false;
ConsumeReturnType returnType = ConsumeReturnType.SUCCESS;
try {
    //还原重试消息的Topic
    ConsumeMessageConcurrentlyService.this.resetRetryTopic(msgs);
    if (msgs != null && !msgs.isEmpty()) {
        for (MessageExt msg : msgs) {
            MessageAccessor.setConsumeStartTimeStamp(msg, String.valueOf(System.currentTimeMillis()));
        }
    }
    //调用Listener的消费逻辑
    status = listener.consumeMessage(Collections.unmodifiableList(msgs), context);
} catch (Throwable e) {
...
}
//统计RT
long consumeRT = System.currentTimeMillis() - beginTimestamp;
/**
 * 处理status：
 * 1、如果为null，抛出异常
 * 2. RT大于15分钟，超时TIME_OUT
 * 3. 重试，返回消费失败
 * 4. 成功，返回成功
 */
if (null == status) {
    if (hasException) {
        returnType = ConsumeReturnType.EXCEPTION;
    } else {
        returnType = ConsumeReturnType.RETURNNULL;
    }
} else if (consumeRT >= defaultMQPushConsumer.getConsumeTimeout() * 60 * 1000) {
    returnType = ConsumeReturnType.TIME_OUT;
} else if (ConsumeConcurrentlyStatus.RECONSUME_LATER == status) {
    returnType = ConsumeReturnType.FAILED;
} else if (ConsumeConcurrentlyStatus.CONSUME_SUCCESS == status) {
    returnType = ConsumeReturnType.SUCCESS;
}
 
if (ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.hasHook()) {
    consumeMessageContext.getProps().put(MixAll.CONSUME_CONTEXT_TYPE, returnType.name());
}
 
if (null == status) {
    log.warn("consumeMessage return null, Group: {} Msgs: {} MQ: {}",
        ConsumeMessageConcurrentlyService.this.consumerGroup,
        msgs,
        messageQueue);
    status = ConsumeConcurrentlyStatus.RECONSUME_LATER;
}
 
if (ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.hasHook()) {
    consumeMessageContext.setStatus(status.toString());
    consumeMessageContext.setSuccess(ConsumeConcurrentlyStatus.CONSUME_SUCCESS == status);
    ConsumeMessageConcurrentlyService.this.defaultMQPushConsumerImpl.executeHookAfter(consumeMessageContext);
}
 
//统计RT
ConsumeMessageConcurrentlyService.this.getConsumerStatsManager()
    .incConsumeRT(ConsumeMessageConcurrentlyService.this.consumerGroup, messageQueue.getTopic(), consumeRT);
 
if (!processQueue.isDropped()) {
    //再次检查，如果未被丢弃了就对结果进行处理
    //处理消费结果：消费成功就返回msgs.size()-1；否则返回-1
    ConsumeMessageConcurrentlyService.this.processConsumeResult(status, context, this);
} else {
    log.warn("processQueue is dropped without process consume result. messageQueue={}, msgs={}", messageQueue, msgs);
}

---

同步调用Broker请求RequestCode.PULL_MESSAGE拉取消息，返回pullResult，这些拉取的消息会存到processqueue中，相当于一个缓存，使用红黑树保存

private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();

            case CLUSTERING:
                //把失败的消息重发会broker，再重消费
                List<MessageExt> msgBackFailed = new ArrayList<MessageExt>(consumeRequest.getMsgs().size());
                for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
                    MessageExt msg = consumeRequest.getMsgs().get(i);
                    boolean result = this.sendMessageBack(msg, context);
                    if (!result) {
                        msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
                        msgBackFailed.add(msg);
                    }
                }
 
                if (!msgBackFailed.isEmpty()) {
                    consumeRequest.getMsgs().removeAll(msgBackFailed);
 
                    this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
                }

@3:ConsumeMessageConcurrentlyService#processConsumeResult

        //移除缓存中的消息
        long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
        //更新消费位点
        if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
            this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
        }