NoHttpResponseException异常分析和优化实践

NoHttpResponseException异常分析和优化实践

在使用HttpClient进行网络请求时，如果服务器端没有响应，可能会抛出NoHttpResponseException异常。该异常表明服务器端没有及时响应，导致客户端无法获取到服务器端的响应。

在实际开发中，我们通常会遇到两种情况：

1. 服务器端没有正常响应，导致客户端无法获取到服务器端的响应。
2. 服务器端响应正常，但客户端无法获取到服务器端的响应。

在实际开发中，我们可以通过以下几种方式来优化NoHttpResponseException异常：

1. 检查服务器端是否正常响应。
   在服务器端，我们可以通过日志来查看是否有异常或错误信息，也可以通过监控工具来查看服务器端的响应情况。
2. 检查客户端是否正常发送请求。
   在客户端，我们可以通过日志来查看是否有异常或错误信息，也可以通过监控工具来查看客户端的请求情况。
3. 检查网络连接是否正常。

现象

线上环境会偶发出现NoHttpResponseException异常，但是不影响业务，可以忽略。

org.apache.http.impl.execchain.RetryExec
I/O exception (java.net.SocketException) caught when processing request to {}->http://xxx.com:80: Connection reset

但是在某些情况下, 会影响线上业务, 比如: 重试多次后还是出现NoHttpResponseException异常:

org.apache.http.NoHttpResponseException: xxx.com:80 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.16.jar!/:4.4.16]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.16.jar!/:4.4.16]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.14.jar!/:4.5.14]

原因分析

排查涉及到的中间件有3个:

1. HttpClient客户端
2. Nginx反向代理
3. 服务器端

在排查系统时, 发现出现异常时, nginx集群正在进行nginx -s reload操作, 导致httpclient客户端无法获取到服务器端的响应。进行抓包分析后发现, nginx在进行reload后及时向客户端发送FIN包, 而客户端(也就是HttpClient客户端)没有响应FIN包, httpclient认为当前连接是正常的, 所以又发送了一个POST请求, 这时候服务器端直接响应了一个RST包,这时候客户端就会抛出NoHttpResponseException异常。

四次挥手:

wireshark抓包数据

解决方案

1. 尽量避免nginx进行reload操作。
2. 避免nginx进行reload操作时, 禁止客户端发送请求。
3. tomcat合理设置keepalive_timeout, 例如: keepalive_timeout: 75000;
4. nginx合理设置keepalive_timeout时间(单位:s), 第一个参数控制nginx连接的存活时间, 第二个是添加到http header里的时间(Keep-Alive: timeout=74), 例如: keepalive_timeout 75 74;
5. http-client合理设置validateAfterInactivity,时间太短容易导致频繁的读取, 时间太长容易导致获取到异常连接. 而且这个参数是一个比较昂贵的操作.

Http Client 配置示例

依赖

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.14</version>
</dependency>

示例

import java.util.concurrent.TimeUnit;
import javax.net.ssl.SSLContext;
import org.apache.http.HttpResponse;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.config.SocketConfig;
import org.apache.http.conn.socket.ConnectionSocketFactory;
import org.apache.http.conn.socket.PlainConnectionSocketFactory;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultConnectionKeepAliveStrategy;
import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.protocol.HttpContext;

public class HttpClientExample {

  public static void main(String[] args) throws ClientProtocolException, IOException {
    CloseableHttpClient httpClient = createHttpClient(1024, 60, 60, 60, 50);
    HttpPost post = new HttpPost("http://www.baidu.com");
    httpClient.execute(post);
    // do some http request
  }

  /**
   * 创建httpclient实例
   * 
   * @param maxTotal
   * @param maxIdleTime 连接空闲时间, 单位: s
   * @param connectTimeToLive 连接存活时间, 单位: s
   * @param keepAliveTimeout keepAlive超时时间, 单位: s
   * @param validateAfterInactivity 连接不活跃需要校验连接的时长, 单位:ms, 默认: 2000ms
   * @return {@link CloseableHttpClient}
   */
  public static CloseableHttpClient createHttpClient(int maxTotal, int maxIdleTime, int connectTimeToLive,
      int keepAliveTimeout, int validateAfterInactivity) {
    int defaultTimeout = 15000;
    RequestConfig defaultRequestConfig = RequestConfig.custom().setConnectTimeout(defaultTimeout)
        .setConnectionRequestTimeout(defaultTimeout).setSocketTimeout(defaultTimeout).build();
    HttpClientBuilder builder = HttpClientBuilder.create().setDefaultRequestConfig(defaultRequestConfig);
    PoolingHttpClientConnectionManager connectionManager =
        createConnectionManager(maxTotal, connectTimeToLive, validateAfterInactivity);
    builder.setConnectionManager(connectionManager);
    builder.setKeepAliveStrategy(new DefaultConnectionKeepAliveStrategy() {
      private long keepAliveTimeoutInner = keepAliveTimeout;

      @Override
      public long getKeepAliveDuration(HttpResponse response, HttpContext context) {
        long keepAliveDuration = super.getKeepAliveDuration(response, context);
        if (keepAliveDuration < 0 && keepAliveTimeoutInner > 0) {
          keepAliveDuration = keepAliveTimeoutInner * 1000;
        }
        return keepAliveDuration;
      }
    });

    builder.evictExpiredConnections().evictIdleConnections(maxIdleTime, TimeUnit.SECONDS)
        .setConnectionTimeToLive(connectTimeToLive, TimeUnit.SECONDS)
        .setRetryHandler(new DefaultHttpRequestRetryHandler());
    return builder.build();
  }

  /**
   * 创建连接池
   * 
   * @param maxTotal 最大连接池数, 默认:20
   * @param connectTimeToLive 连接存活时间, 单位: s
   * @param validateAfterInactivity 连接不活跃需要校验连接的时长, 单位:ms,默认: 2000ms
   * @return {@link PoolingHttpClientConnectionManager}
   */
  public static PoolingHttpClientConnectionManager createConnectionManager(int maxTotal, int connectTimeToLive,
      int validateAfterInactivity) {
    RegistryBuilder<ConnectionSocketFactory> sockRegistryBuilder = RegistryBuilder.<ConnectionSocketFactory>create();
    SSLContext sslContext = SslContextUtil.getSslContext();
    sockRegistryBuilder.register("http", PlainConnectionSocketFactory.INSTANCE);
    sockRegistryBuilder.register("https", new SSLConnectionSocketFactory(sslContext, NoopHostnameVerifier.INSTANCE));
    Registry<ConnectionSocketFactory> sockRegistry = sockRegistryBuilder.build();
    PoolingHttpClientConnectionManager connectionManager =
        new PoolingHttpClientConnectionManager(sockRegistry, null, null, null, connectTimeToLive, TimeUnit.SECONDS);
    if (maxTotal > 0) {
      connectionManager.setMaxTotal(maxTotal);// 默认:20
      connectionManager.setDefaultMaxPerRoute(maxTotal);// 默认: 2
    }
    if (validateAfterInactivity > 0) {
      connectionManager.setValidateAfterInactivity(validateAfterInactivity);// 默认:2000 
    }
    connectionManager.setDefaultSocketConfig(
        SocketConfig.custom().setSoTimeout(HttpConstants.DEFAULT_TIMEOUT).setTcpNoDelay(true).build());
    return connectionManager;
  }

}