五Dubbo服务引用源码分析--4网络处理-4.2provider端处理请求
五Dubbo服务引用源码分析--4网络处理-4.2provider端处理请求
5.3.3 provider端处理请求(请求消息处理)
在前述章节中，知道在NettyServer初始化过程中，向dubbo的底层netty通信框架的pipeline上添加了编解码handler、NettyHandler的channelHandler处理器。
然后，添加的nettyHandler的处理链如下：
NettyHandler->NettyServer->MultiMessageHandler#received -> HeartbeatHandler#received -> Dubbo线程模型指定的 Handler#received -> DecodeHandler#received -> HeaderExchangeHandler#received -> DubboProtocol.requestHandler#received
由内到外的ChanelHandler处理链：
DubboProtocol.ExchangeHandlerAdapter
HeaderExchangeHandler主要还是管理连接等。
DecodeHandler主要是对请求进行解码。
AllChannelHandler是主要负责线程管理。
HeartbeatHandler主要负责心跳检测。
MultiMessageHandler主要负责将Dubbo内部定义的多条消息的聚合消息进行拆分处理。
NettyHandler
因此，当consumer端发送请求后，provider端netty的channel接受到消息后，会调用InternalDecode.decode-->Codec.decode反序列化、解码(解码、反序列化部分暂时不做详细分析)
DubboCountCodec#decode -> DubboCodec#decode -> Serialization#deserialize
然后，会调用配置的nettyHandler处理发送来的消息。大致处理流程如下：
NettyServerHandler#channelRead ->
NettyServer#received ->
MultiMessageHandler#received ->
HeartbeatHandler#received ->
AllChannelHandler#received (根据Dubbo 线程模型的不同会有不同的实现) ->
DecodeHandler#received ->
HeaderExchangeHandler#received -> 判断方法请求，是否需要返回值：
请求需要返回值 : ExchangeHandler#reply (DubboProtocol 内部的 requestHandler 属性)
请求不需返回值 : ExchangeHandler#received (DubboProtocol 内部的 requestHandler 属性) -> ExchangeHandler#received
现在，开始跟进provider端代码处理流程：
TAG1 NettyHandler.messageReceived
nettyHandler是初始化nettyServer时，创建的new NettyHandler(nettyServer)，并传入nettyServer的pipeline的处理器。
NettyHandler.messageReceived
@Override
public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
//channel对象的属性见下面
NettyChannel channel = NettyChannel.getOrAddChannel(ctx.getChannel(), url, handler);
try {
//handler为NettyServer对象,执行消息的接收处理
handler.received(channel, e.getMessage());
} finally {
NettyChannel.removeChannelIfDisconnected(ctx.getChannel());
}
}
nettyChannel的属性如下，其中包含了channelHandler的处理器链
TAG1.1 NettyServer.received------channelHandler链式处理
nettyServer的received方法实现在父类中
AbstractPeer
@Override
public void received(Channel ch, Object msg) throws RemotingException {
if (closed) {
return;
}
//未做处理，执行MultiMessageHandler.received
handler.received(ch, msg);
}
MultiMessageHandler
//主要负责将Dubbo内部定义的多条消息的聚合消息进行拆分处理
@Override
public void received(Channel channel, Object message) throws RemotingException {
if (message instanceof MultiMessage) {
//如果消息是多条的聚合，则转换并逐个接收处理
MultiMessage list = (MultiMessage) message;
for (Object obj : list) {
handler.received(channel, obj);
}
} else {
//HeartbeatHandler
handler.received(channel, message);
}
}
HeartbeatHandler
//处理心跳
public void received(Channel channel, Object message) throws RemotingException {
setReadTimestamp(channel);
//心跳请求的处理
if (isHeartbeatRequest(message)) {
Request req = (Request) message;
if (req.isTwoWay()) { //需要返回值的心跳请求，构造响应response，并同步发送心跳响应
Response res = new Response(req.getId(), req.getVersion());
res.setEvent(Response.HEARTBEAT_EVENT);
channel.send(res);
if (logger.isInfoEnabled()) {
int heartbeat = channel.getUrl().getParameter(Constants.HEARTBEAT_KEY, 0);
if (logger.isDebugEnabled()) {
logger.debug("Received heartbeat from remote channel " + channel.getRemoteAddress()
+ ", cause: The channel has no data-transmission exceeds a heartbeat period"
+ (heartbeat > 0 ? ": " + heartbeat + "ms" : ""));
}
}
}
return;
}
//心跳响应的处理
if (isHeartbeatResponse(message)) {
if (logger.isDebugEnabled()) {
logger.debug("Receive heartbeat response in thread " + Thread.currentThread().getName());
}
return;
}
//TAG1.2 AllChannelHandler.received
handler.received(channel, message);
}
HeartbeatHandler心跳处理器，用来处理心跳请求和心跳响应。如果是正常的request请求，将其转发给下一个channelHandler。
TAG1.2 AllChannelHandler.received(provider的线程模型)--异步处理
在配置AllDispatcher线程模型部分，讲到provider端默认的线程模型为all，即会把所有的消息都派发到业务线程池，这些消息包括请求、响应、连接事件、断开事件、心跳事件等。
AllChannelHandler.received
public void received(Channel channel, Object message) throws RemotingException {
//provider端业务线程池，默认最大、核心线程数为200
ExecutorService cexecutor = getExecutorService();
try {
//将request请求，包装成ChannelEventRunnable，然后投递给线程池处理
cexecutor.execute(new ChannelEventRunnable(channel, handler, ChannelState.RECEIVED, message));
} catch (Throwable t) {
//TODO A temporary solution to the problem that the exception information can not be sent to the opposite end after the thread pool is full. Need a refactoring
//fix The thread pool is full, refuses to call, does not return, and causes the consumer to wait for time out
if(message instanceof Request && t instanceof RejectedExecutionException){
Request request = (Request)message;
if(request.isTwoWay()){
String msg = "Server side(" + url.getIp() + "," + url.getPort() + ") threadpool is exhausted ,detail msg:" + t.getMessage();
Response response = new Response(request.getId(), request.getVersion());
response.setStatus(Response.SERVER_THREADPOOL_EXHAUSTED_ERROR);
response.setErrorMessage(msg);
channel.send(response);
return;
}
}
throw new ExecutionException(message, channel, getClass() + " error when process received event .", t);
}
}
在all线程池模型中，将接收到的request，包装成ChannelEventRunnable对象，设置channelEvent事件类型为ChannelState.RECEIVED。然后交给线程池异步执行request任务。
ChannelEventRunnable
@Override
public void run() {
//处理channel接收请求
if (state == ChannelState.RECEIVED) {
try {
//decodeHandler
handler.received(channel, message);
} catch (Exception e) {
logger.warn("ChannelEventRunnable handle " + state + " operation error, channel is " + channel
+ ", message is " + message, e);
}
} else {
switch (state) {
case CONNECTED:
try {
handler.connected(channel);
} catch (Exception e) {
logger.warn("ChannelEventRunnable handle " + state + " operation error, channel is " + channel, e);
}
break;
case DISCONNECTED:
try {
handler.disconnected(channel);
} catch (Exception e) {
logger.warn("ChannelEventRunnable handle " + state + " operation error, channel is " + channel, e);
}
break;
case SENT:
try {
handler.sent(channel, message);
} catch (Exception e) {
logger.warn("ChannelEventRunnable handle " + state + " operation error, channel is " + channel
+ ", message is " + message, e);
}
case CAUGHT:
try {
handler.caught(channel, exception);
} catch (Exception e) {
logger.warn("ChannelEventRunnable handle " + state + " operation error, channel is " + channel
+ ", message is: " + message + ", exception is " + exception, e);
}
break;
default:
logger.warn("unknown state: " + state + ", message is " + message);
}
}
}
TAG1.3 DecodeHandler
DecodeHandler
@Override
public void received(Channel channel, Object message) throws RemotingException {
if (message instanceof Decodeable) {
decode(message);
}
//对request请求，其内的mdata解码
if (message instanceof Request) {
decode(((Request) message).getData());
}
//TAG1.4 HeaderExchangeHandler
if (message instanceof Response) {
decode(((Response) message).getResult());
}
handler.received(channel, message);
}
此时message.Data类型为DecodeableRpcInvocation
TAG1.4 HeaderExchangeHandler.received---(isTwoWay处理)
HeaderExchangeHandler.received的作用是根据请求request类型，根据是否需要返回值，进行对应的处理逻辑：
其中，exchangeChannel对象属性如下：
HeaderExchangeHandler.received
@Override
public void received(Channel channel, Object message) throws RemotingException {
channel.setAttribute(KEY_READ_TIMESTAMP, System.currentTimeMillis());
//获取HeaderExchangeChannel对象，其内包裹NettyChannel
ExchangeChannel exchangeChannel = HeaderExchangeChannel.getOrAddChannel(channel);
try {
/**………………………………………………………………………………………………………………处理request请求…………………………………………………………………………………… */
if (message instanceof Request) {
// handle request.
Request request = (Request) message;
//处理事件对象，如心跳事件
if (request.isEvent()) {
handlerEvent(channel, request);
} else {
//如果request需要有返回值，此时需要获取返回结果，并返回给consumer端
if (request.isTwoWay()) {
//TAG1.5 handleRequest(exchangeChannel, request)
Response response = handleRequest(exchangeChannel, request);
//TAG1.6 channel.send(response)
channel.send(response);
}
//如果不需要返回值的request，仅仅向后调用指定服务，不需要返回调用结果
else {
//TAG1.7 DubboProtocol#ExchangeHandlerAdapter dubboProtocol初始化创建的属性--也是一个ChannelHandler对象
handler.received(exchangeChannel, request.getData());
}
}
}
/**………………………………………………………………………………………………………………处理response响应…………………………………………………………………………………… */
else if (message instanceof Response) {
handleResponse(channel, (Response) message);
} else if (message instanceof String) {
//telnet相关、dubbo的qos相关请求信息
if (isClientSide(channel)) {
Exception e = new Exception("Dubbo client can not supported string message: " + message + " in channel: " + channel + ", url: " + channel.getUrl());
logger.error(e.getMessage(), e);
} else {
String echo = handler.telnet(channel, (String) message);
if (echo != null && echo.length() > 0) {
channel.send(echo);
}
}
} else {
handler.received(exchangeChannel, message);
}
} finally {
HeaderExchangeChannel.removeChannelIfDisconnected(channel);
}
}
对于双向通信，provider端（接受request）和consumer端（接受response）都会调用HeaderExchangeHandler.received处理接收的消息。所以，这里需要对不同的消息类型，进行对应的处理逻辑---此处仅针对消息类型为request进行分析：
request=(request) message
1 如果request.isEvent------handlerEvent(channel,request);
2 非事件类型的request：
2.1 如果request.isTwoWay--即需要返回值的请求
Response response = handleRequest(exchangeChannel, request);//处理请求，获取结果
channel.send(response);//将返回值response写回通道
2.2 不需要返回值的请求
handler.received(exchangeChannel, request.getData()); //此时仅交给DubboProtocol#ExchangeHandlerAdapter.receive处理
TAG1.5 HeaderExchangeHandler.handleRequest(exchangeChannel, request)--(2.7异步改造)
HeaderExchangeHandler
Response handleRequest(ExchangeChannel channel, Request req) throws RemotingException {
//根据请求request的id创建response对象
Response res = new Response(req.getId(), req.getVersion());
if (req.isBroken()) {
Object data = req.getData();
String msg;
if (data == null) msg = null;
else if (data instanceof Throwable) msg = StringUtils.toString((Throwable) data);
else msg = data.toString();
res.setErrorMessage("Fail to decode request due to: " + msg);
res.setStatus(Response.BAD_REQUEST);
return res;
}
// find handler by message class.
//获取request中的data数据对象
Object msg = req.getData();
try {
// handle data.
//TAG1.5.1 DubboProtocol#ExchangeHandlerAdapter dubboProtocol初始化创建的属性--也是一个ChannelHandler对象
Object result = handler.reply(channel, msg);
res.setStatus(Response.OK);
res.setResult(result);
} catch (Throwable e) {
res.setStatus(Response.SERVICE_ERROR);
res.setErrorMessage(StringUtils.toString(e));
}
return res;
}
msg对象如下：
//TAG1.5.1 DubboProtocol#ExchangeHandlerAdapter.reply--(2.7异步改造)
dubboProtocol初始化创建的属性--也是一个ChannelHandler对象。
public class DubboProtocol extends AbstractProtocol {
//dubboProtocol初始化时，创建的属性对象
private ExchangeHandler requestHandler = new ExchangeHandlerAdapter() {
@Override
public Object reply(ExchangeChannel channel, Object message) throws RemotingException {
if (message instanceof Invocation) {
Invocation inv = (Invocation) message;
//DP1 DubboProtocol.getInvoker 获取被ProtocolFilterWrapper和ProtocolListenerWrapper包装后的invoker
Invoker<?> invoker = getInvoker(channel, inv);
// need to consider backward-compatibility if it's a callback
//如果当前invocation调用的服务时回调服务的处理
if (Boolean.TRUE.toString().equals(inv.getAttachments().get(IS_CALLBACK_SERVICE_INVOKE))) {
String methodsStr = invoker.getUrl().getParameters().get("methods");
boolean hasMethod = false;
if (methodsStr == null || methodsStr.indexOf(",") == -1) {
hasMethod = inv.getMethodName().equals(methodsStr);
} else {
String[] methods = methodsStr.split(",");
for (String method : methods) {
if (inv.getMethodName().equals(method)) {
hasMethod = true;
break;
}
}
}
if (!hasMethod) {
logger.warn(new IllegalStateException("The methodName " + inv.getMethodName()
+ " not found in callback service interface ,invoke will be ignored."
+ " please update the api interface. url is:"
+ invoker.getUrl()) + " ,invocation is :" + inv);
return null;
}
}
//在rpcContext上下文对象中设置远端调用的地址
RpcContext.getContext().setRemoteAddress(channel.getRemoteAddress());
//DP2 invoker.invoke(inv)调用服务
return invoker.invoke(inv);
}
throw new RemotingException(channel, "Unsupported request: "
+ (message == null ? null : (message.getClass().getName() + ": " + message))
+ ", channel: consumer: " + channel.getRemoteAddress() + " --> provider: " + channel.getLocalAddress());
}
DP1 DubboProtocol.getInvoker
获取被ProtocolFilterWrapper和ProtocolListenerWrapper包装后的invoker
Invoker<?> getInvoker(Channel channel, Invocation inv) throws RemotingException {
boolean isCallBackServiceInvoke = false;
boolean isStubServiceInvoke = false;
int port = channel.getLocalAddress().getPort();//20880
String path = inv.getAttachments().get(Constants.PATH_KEY);//com.alibaba.dubbo.demo.DemoService
// if it's callback service on client side
isStubServiceInvoke = Boolean.TRUE.toString().equals(inv.getAttachments().get(Constants.STUB_EVENT_KEY));
if (isStubServiceInvoke) {
port = channel.getRemoteAddress().getPort();
}
//callback 服务是回调类型的invoker
isCallBackServiceInvoke = isClientSide(channel) && !isStubServiceInvoke;
if (isCallBackServiceInvoke) {
path = inv.getAttachments().get(Constants.PATH_KEY) + "." + inv.getAttachments().get(Constants.CALLBACK_SERVICE_KEY);
inv.getAttachments().put(IS_CALLBACK_SERVICE_INVOKE, Boolean.TRUE.toString());
}
//构造存储在DubboProtocol#exporterMap中的serviceKey
//com.alibaba.dubbo.demo.DemoService:20880
String serviceKey = serviceKey(port, path, inv.getAttachments().get(Constants.VERSION_KEY), inv.getAttachments().get(Constants.GROUP_KEY));
/**……………………………………………………………………………………从DubboProtocol缓存表获取exporter……………………………………………………………………………………………………… */
DubboExporter<?> exporter = (DubboExporter<?>) exporterMap.get(serviceKey);
if (exporter == null)
throw new RemotingException(channel, "Not found exported service: " + serviceKey + " in " + exporterMap.keySet() + ", may be version or group mismatch " + ", channel: consumer: " + channel.getRemoteAddress() + " --> provider: " + channel.getLocalAddress() + ", message:" + inv);
return exporter.getInvoker();
}
exporterMap是在服务暴露过程中，添加的缓存
获取到暴露服务对象DubboExporter，其内包括被filter、listener包裹的invoker对象
DP2 invoker.invoke(inv)调用服务
获取的invoker是被listener、filter包裹的invoker。
这里，直接从RegistryProtocol$InvokerDegegete代码处跟入，跳过filter链的拦截处理：
RegistryProtocol
//这个类其到代理作用，将对invoker.invoke调用转发，
public static class InvokerDelegete<T> extends InvokerWrapper<T> {
private final Invoker<T> invoker;
public InvokerDelegete(Invoker<T> invoker, URL url) {
super(invoker, url);
this.invoker = invoker;
}
public Invoker<T> getInvoker() {
if (invoker instanceof InvokerDelegete) {
return ((InvokerDelegete<T>) invoker).getInvoker();
} else {
return invoker;
}
}
}
InvokerDelegete保存invoker与对应的服务地址的关系对
然后跟入DelegateProviderMetaDataInvoker
public Result invoke(Invocation invocation) throws RpcException {
//DP2.1 AbstractProxyInvoker.invoke
return invoker.invoke(invocation);
}
//DP2.1 AbstractProxyInvoker.invoke----JavassistProxyFactory.getInvoker创建的匿名invoker
@Override
public Result invoke(Invocation invocation) throws RpcException {
try {
//doInvoke 由子类实现
return new RpcResult(doInvoke(proxy, invocation.getMethodName(), invocation.getParameterTypes(), invocation.getArguments()));
} catch (InvocationTargetException e) {
return new RpcResult(e.getTargetException());
} catch (Throwable e) {
throw new RpcException("Failed to invoke remote proxy method " + invocation.getMethodName() + " to " + getUrl() + ", cause: " + e.getMessage(), e);
}
}
此处的子类，是JavassistProxyFactory中创建的匿名对象
public class JavassistProxyFactory extends AbstractProxyFactory {
@Override
@SuppressWarnings("unchecked")
public <T> T getProxy(Invoker<T> invoker, Class<?>[] interfaces) {
return (T) Proxy.getProxy(interfaces).newInstance(new InvokerInvocationHandler(invoker));
}
@Override
public <T> Invoker<T> getInvoker(T proxy, Class<T> type, URL url) {
// TODO Wrapper cannot handle this scenario correctly: the classname contains '$'
//使用Wrapper.getWrapper包装代理对象获取的字节码并实例化的实例
final Wrapper wrapper = Wrapper.getWrapper(proxy.getClass().getName().indexOf('$') < 0 ? proxy.getClass() : type);
return new AbstractProxyInvoker<T>(proxy, type, url) {
@Override
protected Object doInvoke(T proxy, String methodName,
Class<?>[] parameterTypes,
Object[] arguments) throws Throwable {
//DP2.1.1 wrapper.invokeMethod
return wrapper.invokeMethod(proxy, methodName, parameterTypes, arguments);
}
};
}
}
//DP2.1.1 wrapper.invokeMethod
这里调用provider端在服务暴露之初，用Wrapper.getWrapper获取的wrapper对象，来代理对象的调用。获取对服务端的真实调用
到这里，对provider端的调用过程就结束了。
总结invoker.invoke的调用流程：
TAG1.6 nettyChannel.send(response)--发送响应到consumer端
此时返回到HeaderExchangeHandler代码中，
上述是provider端调用服务获取的结果
然后调用nettyChannel发送响应消息到consumer端。后续的过程，和consumer端发送request的流程相同
AbstractPeer
public void send(Object message) throws RemotingException {
send(message, url.getParameter(Constants.SENT_KEY, false));
}
NettyChannel
public void send(Object message, boolean sent) throws RemotingException {
super.send(message, sent);
boolean success = true;
int timeout = 0;
try {
//调用netty底层NioAcceptedSocketChannel将响应response写入通道channel中
ChannelFuture future = channel.write(message);
if (sent) {
timeout = getUrl().getPositiveParameter(Constants.TIMEOUT_KEY, Constants.DEFAULT_TIMEOUT);
success = future.await(timeout);
}
Throwable cause = future.getCause();
if (cause != null) {
throw cause;
}
} catch (Throwable e) {
throw new RemotingException(this, "Failed to send message " + message + " to " + getRemoteAddress() + ", cause: " + e.getMessage(), e);
}
if (!success) {
throw new RemotingException(this, "Failed to send message " + message + " to " + getRemoteAddress()
+ "in timeout(" + timeout + "ms) limit");
}
}
服务端，最后调用NettyChannel.send将响应写入通道，发送到consumer端。
