ConnectionPoolTimeoutException: Timeout waiting for connection from pool
背景
今天在通过监控系统发现一个错误,错误如下
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
动作
- 起初怀疑是连接池设置过小,故查看代码对连接池大小的设置
- 发现连接池设置已经足够大,我们的业务完全用不到这么多的连接数,故怀疑连接用完没有被释放。查看服务器上连接数
netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
查看代码, 发现确实没有释放连接,如下
HttpResponse response = httpclient.execute(httpPost);
List<String> lines = IOUtils.readLines(response.getEntity().getContent(), "utf-8");
StringBuffer res = new StringBuffer();
for (String line : lines) {
res.append(line);
}
ChatMessage chatMessage = getChatMessage(sendChatMessageDTO, replyMessageInfo, sendMessageesResultBO);
return chatMessage;
解决
即然发现了问题,我们把连接用完归还到连接池即可。
HttpPost httpPost = new HttpPost(connectUrl);
InputStream inputStream = null;
try {
HttpResponse response = httpclient.execute(httpPost);
inputStream = response.getEntity().getContent();
List<String> lines = IOUtils.readLines(inputStream, "utf-8");
} catch (Exception e){
logger.error("connect error", e);
if(httpPost != null){
httpPost.abort();
}
} finally {
if(inputStream != null){
try {
inputStream.close();
} catch (Exception e){
logger.error("", e);
}
}
}
思考(基于版本4.5)
- httpclient 连接池原理是什么样的呢?
连接池中主要有三个属性,用于不同状态的连接
//存放正在使用的连接
private final Set<E> leased;
//存放可用的连接
private final LinkedList<E> available;
//存放 pending 的连接
private final LinkedList<PoolEntryFuture<E>> pending;
- 默认连接池的大小是多少?
我们查看官方代码(版本4.5), 发现其默认总大小为20,每个 route 的默认最大为 2
public PoolingHttpClientConnectionManager(
final HttpClientConnectionOperator httpClientConnectionOperator,
final HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory,
final long timeToLive, final TimeUnit timeUnit) {
super();
this.configData = new ConfigData();
this.pool = new CPool(new InternalConnectionFactory(
this.configData, connFactory), 2, 20, timeToLive, timeUnit);
this.pool.setValidateAfterInactivity(2000);
this.connectionOperator = Args.notNull(httpClientConnectionOperator, "HttpClientConnectionOperator");
this.isShutDown = new AtomicBoolean(false);
}
- close 的原理是怎样呢?
有两种 close
- CloseableHttpResponse.close(): 这种方式是将用完的连接放回连接池的可用集合,并不是将连接真正关闭。内部调用的是 release()
- httpClient.close(): 这种是关闭连接池,将池中的所有资源释放,即真正关闭连接