org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe问题探究

背景

今天下午遇到同事求助，说是服务端出现了好几个java.io.IOException: Broken pipe这样的异常，让我帮忙看一下，这个问题对于我们做服务端开发的技术人员是很容易遇到的，特此记录一下。

探究

问题堆栈

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:356)
    at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:825)
    at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:730)
    at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:391)
    at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:369)
    at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
    at org.springframework.session.web.http.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(    OnCommittedResponseWrapper.java:620)
    at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(   OnCommittedResponseWrapper.java:639)
    at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(   OnCommittedResponseWrapper.java:639)
    at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085)
    at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeBytes(UTF8JsonGenerator.java:1173)
    at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeFieldName(UTF8JsonGenerator.java:256)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:725)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
    at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
    at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
    at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
    at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
    at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913)
    at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(  AbstractJackson2HttpMessageConverter.java:286)
    at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:102)
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(    AbstractMessageConverterMethodProcessor.java:272)
    at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(  RequestResponseBodyMethodProcessor.java:180)
    at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(   HandlerMethodReturnValueHandlerComposite.java:82)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle( ServletInvocableHandlerMethod.java:119)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(  RequestMappingHandlerAdapter.java:877)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(   RequestMappingHandlerAdapter.java:783)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)    

着重关注点

堆栈信息中有一行比较关键，在OutputBuffer :: realWriteBytes这个方法中出现了异常，在方法内部抛出了ClientAbortException：　

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:356)

OutputBuffer#realWriteBytes

/**
 * Sends the buffer data to the client output, checking the
 * state of Response and calling the right interceptors.
 *
 * @param buf the ByteBuffer to be written to the response
 *
 * @throws IOException An underlying IOException occurred
 */
public void realWriteBytes(ByteBuffer buf) throws IOException {
 
    if (closed) {
        return;
    }
    if (coyoteResponse == null) {
        return;
    }
 
    // If we really have something to write
    if (buf.remaining() > 0) {
        // real write to the adapter
        try {
            coyoteResponse.doWrite(buf);
        } catch (IOException e) {
            // An IOException on a write is almost always due to
            // the remote client aborting the request. Wrap this
            // so that it can be handled better by the error dispatcher.
            throw new ClientAbortException(e);
        }
    }
 
}

在上面的代码中，调用coyoteResponse.doWrite(buf);方法时，会抛出ClientAbortException异常

我们仔细阅读以下上述注释：

An IOException on a write is almost always due to the remote client aborting the request.
Wrap this so that it can be handled better by the error dispatcher.
 
进行写操作时出现的IOException几乎总是由于远程客户端中止请求而导致的。将其包装起来，以便错误调度程序可以更好地处理它。　

结论

有位大神将此问题对应的流程图画的很好，特摘录过来：秋夜无霜

如上图：

调用方通过restTemplate封装的http连接池，这时候连接设置connectTimeout=3000,readTimeout=3000，然后网络请求http请求+端口调用到服务方，然后调用一个api（uri），这时候响应的时候，然后Spring框架对响应进行JSON序列化，然后写入Socket，这时候调用方（客户端）突然Socket端中断连接，而Server端正在进行输出流写，导致如上异常问题。

然后针对当前业务场景，看了今天300个请求，有不到10个出现这种问题，超时大多3、4秒样子，我们通知调用方增大连接池的超时时间，就不再出现这个问题。

最近气温变化大，冷暖转换明显，由于没有及时增添衣物，注意保暖，翎野君感染风寒，现在正在调养恢复中，希望大家也多多注意，春捂秋冻。

参考文章

https://blog.csdn.net/shichen2010/article/details/121769535