netty处理拆包
https://blog.csdn.net/u010853261/article/details/55803933
[netty]--最通用TCP黏包解决方案:LengthFieldBasedFrameDecoder和LengthFieldPrepender
前面已经说过:
TCP以流的方式进行数据传输,上层应用协议为了对消息进行区分,往往采用如下4种方式。
(1)消息长度固定:累计读取到固定长度为LENGTH之后就认为读取到了一个完整的消息。然后将计数器复位,重新开始读下一个数据报文。(2)回车换行符作为消息结束符:在文本协议中应用比较广泛。
(3)将特殊的分隔符作为消息的结束标志,回车换行符就是一种特殊的结束分隔符。
(4)通过在消息头中定义长度字段来标示消息的总长度。
netty中针对这四种场景均有对应的解码器作为解决方案,比如:
(1)通过FixedLengthFrameDecoder 定长解码器来解决定长消息的黏包问题;
(2)通过LineBasedFrameDecoder和StringDecoder来解决以回车换行符作为消息结束符的TCP黏包的问题;
(3)通过DelimiterBasedFrameDecoder 特殊分隔符解码器来解决以特殊符号作为消息结束符的TCP黏包问题;
(4)最后一种,也是本文的重点,通过LengthFieldBasedFrameDecoder 自定义长度解码器解决TCP黏包问题。
大多数的协议在协议头中都会携带长度字段,用于标识消息体或则整包消息的长度。LengthFieldBasedFrameDecoder通过指定长度来标识整包消息,这样就可以自动的处理黏包和半包消息,只要传入正确的参数,就可以轻松解决“读半包”的问题。
https://blog.csdn.net/bestone0213/article/details/47108419
netty处理粘包问题——1
发送数据是: | ABC | DEF | GHI |
+-----+-----+-----+
- public byte[] send(byte[] sendData) throws UnknownHostException, IOException {
- Socket socket = new Socket(serverIp, serverPort);
- OutputStream os = socket.getOutputStream();
- InputStream is = socket.getInputStream();
- byte resultArray[] = null;
- try {
- // 定义一个发送消息协议格式:|--header:4 byte--|--content:10MB--|
- // 获取一个4字节长度的协议体头
- byte[] dataLength = intToByteArray(4, sendData.length);
- // 和请求的数据组成一个请求数据包
- byte[] requestMessage = combineByteArray(dataLength, sendData);
- //发送数据-------------------------------
- os.write(requestMessage);
- os.flush();
- //接收数据-------------------------------
- resultArray = IOUtils.toByteArray(is);
- } catch (Exception e) {
- e.printStackTrace();
- } finally {
- os.close();
- is.close();
- socket.close();
- }
- return resultArray;
- }
- private static byte[] intToByteArray(int byteLength, int intValue) {
- return ByteBuffer.allocate(byteLength).putInt(intValue).array();
- }
- private static byte[] combineByteArray(byte[] array1, byte[] array2) {
- byte[] combined = new byte[array1.length + array2.length];
- System.arraycopy(array1, 0, combined, 0, array1.length);
- System.arraycopy(array2, 0, combined, array1.length, array2.length);
- return combined;
- }
- @Override
- public void initChannel(SocketChannel ch) throws Exception {
- ChannelPipeline pipeline = ch.pipeline();
- pipeline.addLast("framedecoder",new LengthFieldBasedFrameDecoder(1024*1024*1024, 0, 4,0,4));
- pipeline.addLast(new TCPServiceHandler());// 处理业务Handler
- }
三、总结:客户端和服务端定义消息格式必须一致
1. frame包整体功能描述
此包主要作用于对TCP/IP数据包的分包和包重组,常用于数据的流传输,是扩展的解码器。
包目录结构如下:
![netty 数据分包、组包、粘包处理机制(一) - 断鸿零雁 - 断鸿零雁的博客](http://img9.ph.126.net/RXdFcbzyshwirEUgEoHRsQ==/6597528263005251319.jpg)
2. 包中各类功能详解
(1) FrameDecoder
抽象类,将ChannelBuffers中的二进制数据转换成有意义的数据帧(frame)对象,一般不直接调用,提供给此包中的FixedLengthFrameDecoder类、DelimiterBasedFrameDecoder类和LengthFieldBasedFrameDecoder类使用,也可以提供给其他类使用(暂不探讨);
在数据传输中,我们发送的数据包如下所示
+-----+-----+-----+
| ABC | DEF | GHI |
+-----+-----+-----+
而实际接收的包的格式为:
+----+-------+---+---+
| AB | CDEFG | H | I |
+----+-------+---+---+
产生的原因为:数据在传输过程中,产生数据包碎片(TCP/IP数据传输时大数据包无法一次传输,被拆分成小数据包,小数据包即为数据包碎片),这就造成了实际接收的数据包和发送的数据包不一致的情况。
而通过FrameDecoder即可实现对上述接收到的数据包的整理,重新还原成如下格式:
+-----+-----+-----+
| ABC | DEF | GHI |
+-----+-----+-----+
如下是一个自定义的Decoder类
public class MyFrameDecoder extends FrameDecoder {
@Override
protected Object decode(ChannelHandlerContext ctx,
ChannelBuffer buf) throws Exception {
// Make sure if the length field was received.
if (buf.readableBytes() < 4) {
// The length field was not received yet - return null.
// This method will be invoked again when more packets are
// received and appended to the buffer.
return null;
}
// The length field is in the buffer.
// Mark the current buffer position before reading the length field
// because the whole frame might not be in the buffer yet.
// We will reset the buffer position to the marked position if
// there's not enough bytes in the buffer.
buf.markReaderIndex();
// Read the length field.
int length = buf.readInt();
// Make sure if there's enough bytes in the buffer.
if (buf.readableBytes() < length) {
// The whole bytes were not received yet - return null.
// This method will be invoked again when more packets are
// received and appended to the buffer.
// Reset to the marked position to read the length field again
// next time.
buf.resetReaderIndex();
return null;
}
// There's enough bytes in the buffer. Read it.
ChannelBuffer frame = buf.readBytes(length);
// Successfully decoded a frame. Return the decoded frame.
return frame;
}
}
此时,我们无需关注数据包是如何重组的,只需要做简单的验证(按照一个包验证)就可以了,FrameDecoder内部实现了组包的机制,不过,此时,需在数据的最前面封装整个数据的长度,示例中数据长度占了四个字节,即前四个字节是数据长度,后面的才是真实的数据。
(2) FixedLengthFrameDecoder
FixedLengthFrameDecoder主要是将诸如
+----+-------+---+---+
| AB | CDEFG | H | I |
+----+-------+---+---+
此类的数据包按照指定的frame长度重新组包,比如确定长度为3,则组包为
+-----+-----+-----+
| ABC | DEF | GHI |
+-----+-----+-----+
构造方法为:new FixedLengthFrameDecoder(int frameLength);
frameLength即修正后的帧长度
另一个构造方法为new FixedLengthFrameDecoder(int frameLength, boolean allocateFullBuffer);
allocateFullBuffer如果为真,则表示初始化的ChannelBuffer大小为frameLength。
(3) Delimiters
分隔符类,DelimiterBasedFrameDecoder类的辅助类。
对Flash XML的socket通信采用nulDelimiter()方法,对于一般的文本采用lineDelimiter()方法
(4) DelimiterBasedFrameDecoder
对接收到的ChannelBuffers按照指定的分隔符Delimiter分隔,分隔符可以是一个或者多个
如将以下数据包按照“\n”分隔:
+--------------+
| ABC\nDEF\r\n |
+--------------+
即为:
+-----+-----+
| ABC | DEF |
+-----+-----+
而如果按照“\r\n”分隔,则为:
+----------+
| ABC\nDEF |
+----------+
对于DelimiterBasedFrameDecoder中的构造方法,其中一些参数说明:
maxFrameLength:解码的帧的最大长度
stripDelimiter:解码时是否去掉分隔符
failFast:为true,当frame长度超过maxFrameLength时立即报TooLongFrameException异常,为false,读取完整个帧再报异常
delimiter:分隔符
(5) LengthFieldBasedFrameDecoder
常用的处理大数据分包传输问题的解决类,先对构造方法LengthFieldBasedFrameDecoder中的参数做以下解释说明“
maxFrameLength:解码的帧的最大长度
lengthFieldOffset :长度属性的起始位(偏移位),包中存放有整个大数据包长度的字节,这段字节的其实位置
lengthFieldLength:长度属性的长度,即存放整个大数据包长度的字节所占的长度
lengthAdjustmen:长度调节值,在总长被定义为包含包头长度时,修正信息长度。initialBytesToStrip:跳过的字节数,根据需要我们跳过lengthFieldLength个字节,以便接收端直接接受到不含“长度属性”的内容
failFast :为true,当frame长度超过maxFrameLength时立即报TooLongFrameException异常,为false,读取完整个帧再报异常
下面对各种情况分别描述:
1. 2 bytes length field at offset 0, do not strip header
lengthFieldOffset = 0
lengthFieldLength = 2
lengthAdjustment = 0
initialBytesToStrip = 0 (= do not strip header)
BEFORE DECODE (14 bytes) AFTER DECODE (14 bytes)
+--------+----------------+ +--------+----------------+
| Length | Actual Content |---->| Length | Actual Content |
| 0x000C | "HELLO, WORLD" | | 0x000C | "HELLO, WORLD" |
+--------+----------------+ +--------+----------------+
此时数据格式不做任何改变(没有跳过任何字节)
2. 2 bytes length field at offset 0, strip header
lengthFieldOffset = 0
lengthFieldLength = 2
lengthAdjustment = 0
initialBytesToStrip = 2 (= the length of the Length field)
BEFORE DECODE (14 bytes) AFTER DECODE (12 bytes)
+--------+----------------+ +----------------+
| Length | Actual Content |---->| Actual Content |
| 0x000C | "HELLO, WORLD" | | "HELLO, WORLD" |
+--------+----------------+ +----------------+
此时帧长度为14个字节,但由于前(lengthFieldOffset = 0)两个(lengthFieldLength = 2)字节是表示帧长度的字节,不计入数据,故真实的数据长度为12个字节。
3. 2 bytes length field at offset 0, do not strip header, the length field represents the length of the whole message
lengthFieldOffset = 0
lengthFieldLength = 2
lengthAdjustment = -2 (= the length of the Length field)
initialBytesToStrip = 0
BEFORE DECODE (14 bytes) AFTER DECODE (14 bytes)
+--------+----------------+ +--------+----------------+
| Length | Actual Content |---->| Length | Actual Content |
| 0x000E | "HELLO, WORLD" | | 0x000E | "HELLO, WORLD" |
+--------+----------------+ +--------+----------------+
此处定义的Length为0x000E共占了两个字节,表示的帧长度为14个字节,前(lengthFieldOffset = 0)两个(lengthFieldLength = 2)字节为Length,由于设置的lengthAdjustment = -2 (= the length of the Length field),故修正的信息实际长度补2,即解码时往前推2个字节,解码后还是14个字节长度(此种情况是把整个长度封装,一般来讲,我们只封装数据长度)
4. 3 bytes length field at the end of 5 bytes header, do not strip header
lengthFieldOffset = 2 (= the length of Header 1)
lengthFieldLength = 3
lengthAdjustment = 0
initialBytesToStrip = 0
BEFORE DECODE (17 bytes) AFTER DECODE (17 bytes)
+---------+---------+--------------+ +---------+---------+------------+
| Header 1| Length |Actual Content|--->| Header 1| Length | Actual Content|
| 0xCAFE | 0x00000C|"HELLO, WORLD"| | 0xCAFE |0x00000C| "HELLO, WORLD"|
+---------+---------+--------------+ +----------+--------+-----------+
此处lengthFieldOffset = 2,从第3个字节开始表示数据长度,长度占3个字节,真实数据长度为0x00000C 即12个字节,而lengthAdjustment=0,initialBytesToStrip = 0,故解码后的数据与解码前的数据相同。
4. 3 bytes length field at the beginning of 5 bytes header, do not strip header
lengthFieldOffset = 0
lengthFieldLength = 3
lengthAdjustment = 2 (= the length of Header 1)
initialBytesToStrip = 0
BEFORE DECODE (17 bytes) AFTER DECODE (17 bytes)
+----------+----------+----------------+ +----------+----------+----------------+
| Length | Header 1 | Actual Content |----->| Length | Header 1 | Actual Content |
| 0x00000C | 0xCAFE | "HELLO, WORLD" | | 0x00000C | 0xCAFE | "HELLO, WORLD" |
+----------+----------+----------------+ +----------+----------+----------------+
此处由于修正的字节数是2,且initialBytesToStrip = 0,故整个数据的解码数据保持不变
总字节数是17,开始的三个字节表示字节长度:12,修正的字节是2,(即从第三个字节开始,再加两个开始是真正的数据,其中跳过的字节数是0)
5. 2 bytes length field at offset 1 in the middle of 4 bytes header, strip the first header field and the length field
lengthFieldOffset = 1 (= the length of HDR1)
lengthFieldLength = 2
lengthAdjustment = 1 (= the length of HDR2)
initialBytesToStrip = 3 (= the length of HDR1 + LEN)
BEFORE DECODE (16 bytes) AFTER DECODE (13 bytes)
+------+--------+------+----------------+ +------+----------------+
| HDR1 | Length | HDR2 | Actual Content |----->| HDR2 | Actual Content |
| 0xCA | 0x000C | 0xFE | "HELLO, WORLD" | | 0xFE | "HELLO, WORLD" |
+------+--------+------+----------------+ +------+----------------+
从第2个字节开始解码,取两个字节作为帧长度,为12个字节,然后,修正一个字节,从第5个字节到最后表示帧数据,解码时,由于initialBytesToStrip=3,表示跳过前三个字节(去掉),故从第四个字节开始解析,解析出来后,如右图所示。
6. 2 bytes length field at offset 1 in the middle of 4 bytes header, strip the first header field and the length field, the length field represents the length of the whole message
lengthFieldOffset = 1
lengthFieldLength = 2
lengthAdjustment = -3 (= the length of HDR1 + LEN, negative)
initialBytesToStrip = 3
BEFORE DECODE (16 bytes) AFTER DECODE (13 bytes)
+------+--------+------+----------------+ +------+----------------+
| HDR1 | Length | HDR2 | Actual Content |----->| HDR2 | Actual Content |
| 0xCA | 0x0010 | 0xFE | "HELLO, WORLD" | | 0xFE | "HELLO, WORLD" |
+------+--------+------+----------------+ +------+----------------+
从第二个字节开始,取两个字节作为帧长度,为16个字节,然后补3个字节,故往前找三个字节,从HDP1开始解码,而又因为initialBytesToStrip=3,解码时忽略掉前三个字节,故从第四个字节开始解析,解析结果如右图所示。
总结:一般来讲,当lengthAdjustment 为负数时,Length表示的是整个帧的长度,当lengthAdjustment为正数或0时,表示真实数据长度。
(6) LengthFieldPrepender
编码类,自动将
+----------------+
| "HELLO, WORLD" |
+----------------+
格式的数据转换成
+--------+----------------+
+ 0x000C | "HELLO, WORLD" |
+--------+----------------+
格式的数据,
如果lengthIncludesLengthFieldLength设置为true,则编码为
+--------+----------------+
+ 0x000E | "HELLO, WORLD" |
+--------+----------------+
格式的数据
应用场景:自定义pipelineFactory类: MyPipelineFactory implements ChannelPipelineFactory
中
pipeline.addLast("frameEncode", new LengthFieldPrepender(4, false));
(7) TooLongFrameException
定义的数据包超过预定义大小异常类
(8) CorruptedFrameException
定义的数据包损坏异常类
3. frame包应用demo
解决分包问题,通常配置MyPipelineFactory中设置,示例如下:
public class MyPipelineFactory implements ChannelPipelineFactory {
@Override
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = Channels.pipeline();
pipeline.addLast("decoder", new LengthFieldBasedFrameDecoder(Integer.MAX_VALUE, 0, 4, 0, 4));
pipeline.addLast("encoder", new LengthFieldPrepender(4, false));
pipeline.addLast("handler", new MyHandler());
return pipeline;
}
}
在客户端设置pipeline.addLast("encoder", new LengthFieldPrepender(4, false));
pipeline.addLast("handler", new MyHandler());
前四个字节表示真实的发送的数据长度Length,编码时会自动加上;
在服务器端设置pipeline.addLast("decoder", new LengthFieldBasedFrameDecoder(Integer.MAX_VALUE, 0, 4, 0, 4));
真实数据最大字节数为Integer.MAX_VALUE,解码时自动去掉前面四个字节