Rocketmq源码解读之消息拉取
最近阅读了Rocketmq关于pullmessage的实现方式，分享出来
众所周知，Rocketmq在consumer端是拉取消息的方式，它会在客户端维护一个PullRequestQueue，这个是一个阻塞队列（LinkedBlockingQueue），内部的节点是PullRequest，每一个PullRequest代表了一个消费的分组单元
PullRequest会记录一个topic对应的consumerGroup的拉取进度，包括MessageQueue和PorcessQueue，还有拉取的offset
（代码片段一）
public class PullRequest { private String consumerGroup; private MessageQueue messageQueue; private ProcessQueue processQueue; private long nextOffset; private boolean lockedFirst = false; }
其中MessageQueue记录元信息：
（代码片段二）
public class MessageQueue implements Comparable<MessageQueue>, Serializable { private String topic; private String brokerName; private int queueId; }
PorcessQueue记录一次拉取之后实际消息体和拉取相关操作记录的快照
（代码片段三）
public class ProcessQueue { private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock(); private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>(); private final AtomicLong msgCount = new AtomicLong(); private final AtomicLong msgSize = new AtomicLong(); private final Lock lockConsume = new ReentrantLock(); private final TreeMap<Long, MessageExt> consumingMsgOrderlyTreeMap = new TreeMap<Long, MessageExt>(); private final AtomicLong tryUnlockTimes = new AtomicLong(0); private volatile long queueOffsetMax = 0L; private volatile boolean dropped = false; private volatile long lastPullTimestamp = System.currentTimeMillis(); private volatile long lastConsumeTimestamp = System.currentTimeMillis(); private volatile boolean locked = false; private volatile long lastLockTimestamp = System.currentTimeMillis(); private volatile boolean consuming = false; private volatile long msgAccCnt = 0; }
PullMessageService负责轮询PullRequestQueue，并进行消息元的拉取
（代码片段四）
public class PullMessageService extends ServiceThread { private final InternalLogger log = ClientLogger.getLog(); private final LinkedBlockingQueue<PullRequest> pullRequestQueue = new LinkedBlockingQueue<PullRequest>(); private final MQClientInstance mQClientFactory; @Override public void run() { while (!this.isStopped()) { try { PullRequest pullRequest = this.pullRequestQueue.take(); this.pullMessage(pullRequest); } catch (InterruptedException ignored) { } catch (Exception e) { log.error("Pull Message Service Run Method exception", e); } } } }
在发送的时候会维护一个PullCallback，这是拉取收到消息后的回调处理
（代码片段五）
public interface PullCallback { void onSuccess(final PullResult pullResult); void onException(final Throwable e); }
这里的实现逻辑就不贴了，本质上就是把消息丢给消费线程池来处理
pullMessage分为同步拉取和异步拉取两种模式，先解读异步拉取，然后再解读同步拉取，再说明两者的区别
其实从这里已经大概可以看出来，异步的方式，这个方法返回值是null，同步的方式必须要返回PullResult，后续说明区别
（代码片段六）
public PullResult pullMessage( final String addr, final PullMessageRequestHeader requestHeader, final long timeoutMillis, final CommunicationMode communicationMode, final PullCallback pullCallback ) throws RemotingException, MQBrokerException, InterruptedException { RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.PULL_MESSAGE, requestHeader); switch (communicationMode) { case ONEWAY: assert false; return null; case ASYNC: this.pullMessageAsync(addr, request, timeoutMillis, pullCallback); return null; case SYNC: return this.pullMessageSync(addr, request, timeoutMillis); default: assert false; break; } return null; }
先介绍异步拉取
可以看到，把PullCallback传进去，并封装了InvokeCallback，
（代码片段七）
private void pullMessageAsync( final String addr, final RemotingCommand request, final long timeoutMillis, final PullCallback pullCallback ) throws RemotingException, InterruptedException { this.remotingClient.invokeAsync(addr, request, timeoutMillis, new InvokeCallback() { @Override public void operationComplete(ResponseFuture responseFuture) { RemotingCommand response = responseFuture.getResponseCommand(); if (response != null) { try { PullResult pullResult = MQClientAPIImpl.this.processPullResponse(response); assert pullResult != null; pullCallback.onSuccess(pullResult); } catch (Exception e) { pullCallback.onException(e); } } else { if (!responseFuture.isSendRequestOK()) { pullCallback.onException(new MQClientException("send request failed to " + addr + ". Request: " + request, responseFuture.getCause())); } else if (responseFuture.isTimeout()) { pullCallback.onException(new MQClientException("wait response from " + addr + " timeout :" + responseFuture.getTimeoutMillis() + "ms" + ". Request: " + request, responseFuture.getCause())); } else { pullCallback.onException(new MQClientException("unknown reason. addr: " + addr + ", timeoutMillis: " + timeoutMillis + ". Request: " + request, responseFuture.getCause())); } } } }); }
接下来进入NettyRemotingAbstract这个类中，使用netty的Chanle发送
（代码片段八）
public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis, final InvokeCallback invokeCallback) throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException { final int opaque = request.getOpaque(); boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS); if (acquired) { final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync); final ResponseFuture responseFuture = new ResponseFuture(opaque, timeoutMillis, invokeCallback, once); this.responseTable.put(opaque, responseFuture); try { channel.writeAndFlush(request).addListener(new ChannelFutureListener() { @Override public void operationComplete(ChannelFuture f) throws Exception { if (f.isSuccess()) { responseFuture.setSendRequestOK(true); return; } else { responseFuture.setSendRequestOK(false); } responseFuture.putResponse(null); responseTable.remove(opaque); try { executeInvokeCallback(responseFuture); } catch (Throwable e) { log.warn("excute callback in writeAndFlush addListener, and callback throw", e); } finally { responseFuture.release(); } log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel)); } }); } catch (Exception e) { responseFuture.release(); log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e); throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e); } } else { if (timeoutMillis <= 0) { throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast"); } else { String info = String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d", timeoutMillis, this.semaphoreAsync.getQueueLength(), this.semaphoreAsync.availablePermits() ); log.warn(info); throw new RemotingTimeoutException(info); } } }
这里需要详细解读一下：
RemotingCommand是request和response的载体，先从request中取出opaque，这是一个自增的操作id，然后将传进来的opaque和invokeCallback封装成一个ResponseFuture，再put到一个叫
responseTable的map中，这个map是一个核心的map，维护着opaque和对应的ResponseFuture
/** * This map caches all on-going requests. */ protected final ConcurrentMap<Integer /* opaque */, ResponseFuture> responseTable = new ConcurrentHashMap<Integer, ResponseFuture>(256);
从注释中可以看出，它缓存着正在执行的request
再回到刚刚的（代码片段八）中，channel.writeAndFlush(request).addListener(new ChannelFutureListener(){...})，netty在writeAndFlush发送完之后会回调我们ChannelFutureListener的operationComplete方法：如果发送成功则responseFuture.setSendRequestOK(true); 并且就return了；如果发送失败，则从responseTable中移除，并且起一个异步线程执行responseFuture中的InvokeCallback，在（代码片段七）中，可以看到当responseFuture.isSendRequestOK()是false的时候，执行了onException，这里就不多介绍了。
那么此时发送的逻辑就全部结束了，整个过程没有任何的阻塞，当Broker收到拉取请求后，会按照queueOffset等信息封装好返回consumer端，
会经过NettyRemotingServer上注册的NettyServerHandler
（代码片段九）
class NettyServerHandler extends SimpleChannelInboundHandler<RemotingCommand> { @Override protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception { processTunnelId(ctx, msg); processMessageReceived(ctx, msg); } public void processTunnelId(ChannelHandlerContext ctx, RemotingCommand msg) { if (nettyServerConfig.isValidateTunnelIdFromVtoaEnable()) { if (null != msg && msg.getType() == RemotingCommandType.REQUEST_COMMAND) { Vtoa vtoa = tunnelTable.get(ctx.channel()); if (null == vtoa) { vtoa = VpcTunnelUtils.getInstance().getTunnelID(ctx); tunnelTable.put(ctx.channel(), vtoa); } msg.addExtField(VpcTunnelUtils.PROPERTY_VTOA_TUNNEL_ID, String.valueOf(vtoa.getVid())); } } } }
最终会调用到NettyRemotingAbstract的processResponseCommand，RemotingCommand中根据opaque从responseTable中获取ResponseFuture，然后同样也是执行callback，这样，就实现了整个pullmessage的异步模式
(代码片段十)
public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) { final int opaque = cmd.getOpaque(); final ResponseFuture responseFuture = responseTable.get(opaque); if (responseFuture != null) { responseFuture.setResponseCommand(cmd); responseTable.remove(opaque); if (responseFuture.getInvokeCallback() != null) { //异步 executeInvokeCallback(responseFuture); //执行回调 } else { //同步 responseFuture.putResponse(cmd); //为了解除阻塞 responseFuture.release(); } } else { log.warn("receive response, but not matched any request, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel())); log.warn(cmd.toString()); } }
我们再看下同步的方式是如何实现的
回顾下代码片段六，同步的方式是需要返回PullResult的，换句话说，这种方式是需要在发送的线程中来处理返回结果的
我们从代码片段六跟下去，跟到NettyRemotingAbstract的invokeSyncImpl
（代码片段十一）
public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis) throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException { final int opaque = request.getOpaque(); try { final ResponseFuture responseFuture = new ResponseFuture(opaque, timeoutMillis, null, null); this.responseTable.put(opaque, responseFuture); final SocketAddress addr = channel.remoteAddress(); channel.writeAndFlush(request).addListener(new ChannelFutureListener() { @Override public void operationComplete(ChannelFuture f) throws Exception { if (f.isSuccess()) { responseFuture.setSendRequestOK(true); return; } else { responseFuture.setSendRequestOK(false); } responseTable.remove(opaque); responseFuture.setCause(f.cause()); responseFuture.putResponse(null); log.warn("send a request command to channel <" + addr + "> failed."); } }); RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis); if (null == responseCommand) { if (responseFuture.isSendRequestOK()) { throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis, responseFuture.getCause()); } else { throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause()); } } return responseCommand; } finally { this.responseTable.remove(opaque); } }
和异步发送的代码片段八对比一下，可以看到，同步方式也要放到responseTable中，这里就有个疑惑了，既然都同步了，还要放到responseTable中干什么呢，继续往下看，
ChannelFutureListener都是一样的，如果发送成功就返回了，然后到了最关键的一行：
RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
这一行顾名思义就是阻塞了，但是也不能一直阻塞住（因为PullMessageService是单线程的，如果因为一个异常就阻塞那就跪了），所以是一个设置了超时时间的阻塞，看下是如何阻塞的
ResponseFuture中有这两个方法，当putResponse的时候，把RemotingCommand赋值，并且countDownLatch.countDown，而在waitResponse的时候countDownLatch.await
（代码片段十二）
public RemotingCommand waitResponse(final long timeoutMillis) throws InterruptedException { this.countDownLatch.await(timeoutMillis, TimeUnit.MILLISECONDS); return this.responseCommand; } public void putResponse(final RemotingCommand responseCommand) { this.responseCommand = responseCommand; this.countDownLatch.countDown(); }
这样就清晰多了，剩下的疑问就是，在什么时候putResponse的，有两个地方：
第一个地方，当拉取消息回来的时候，回顾下代码片段十，有一句是（responseFuture.getInvokeCallback() != null），通过刚刚的流程已经知道，只有异步的时候invokeCallback才不为null，因此走到else，看到在这个时候responseFuture.putResponse(cmd)和responseFuture.release()，也就是说同步方式也是通过responseTable存储的方式，来获取结果，并且通过CountDownLatch来阻塞发送的线程，当收到消息之后再countDown，发送端最终返回PullResult来处理消息
第二个地方，回顾下代码片段十一，在ChannelFutureListener中当发送失败了以后，也会put一个null值：responseFuture.putResponse(null)，这里只是为了将阻塞放开
至此，Rockmq关于pullmessage的同步和异步方式就已经说明白了，总结一下，同步和异步本质上都是“异步”的，因为netty就是一个异步的框架，Rockmq只是利用了CountDownLatch来阻塞住发送端线程来实现了“同步”的效果，
通过一个responseTable来缓存住发送出去的请求，等收到的时候从这个缓存里按对应关系取出来，再去做对应的consumer线程的消息处理
