Apache HttpClient和HttpAsyncClient应用
官网地址 http://hc.apache.org/
官方PDF地址: http://hc.apache.org/httpcomponents-core-ga/tutorial/pdf/httpcore-tutorial.pdf
HttpClient是基于HttpCore的HTTP / 1.1兼容HTTP代理实现。 它还为客户端身份验证,HTTP状态管理和HTTP连接管理提供可重用的组件。 HttpComponents Client是Commons HttpClient 3.x的继承者和替代者。 强烈建议Commons HttpClient用户进行升级。
HttpClient 特性
- 基于标准的纯Java,HTTP版本1.0和1.1的实现
- 在可扩展的OO框架中完全实现所有HTTP方法(GET,POST,PUT,DELETE,HEAD,OPTIONS和TRACE)。
- 支持使用HTTPS(基于SSL的HTTP)协议进行加密。
- 通过HTTP代理的透明连接。
- 通过CONNECT方法通过HTTP代理建立的HTTPS连接隧道。
- 基本,摘要,NTLMv1,NTLMv2,NTLM2会话,SNPNEGO,Kerberos身份验证方案。
- 用于自定义身份验证方案的插件机制。
- 可插拔的安全套接字工厂,使使用第三方解决方案更加容易
- 在多线程应用程序中使用的连接管理支持。支持设置
- 最大总连接数以及每个主机的最大连接数。检测并关闭陈旧的连接。
- 自动Cookie处理,用于从服务器读取Set-Cookie:标头,并在适当时在Cookie:标头中发回。
- 自定义Cookie策略的插件机制。
- 请求输出流,以避免通过直接流到服务器的套接字来缓冲任何内容主体。
- 响应输入流,通过直接从套接字流传输到服务器来有效读取响应主体。
- 在HTTP / 1.0中使用KeepAlive的持久连接和在HTTP / 1.1中的持久性
- 直接访问服务器发送的响应代码和标头。
- 设置连接超时的能力。
- 支持HTTP / 1.1响应缓存。
Asynch HttpClient是基于HttpCore NIO和HttpClient组件的HTTP / 1.1兼容HTTP代理实现。 它是Apache HttpClient的补充模块,适用于特殊情况,在特殊情况下,就原始数据吞吐量而言,处理大量并发连接的能力比性能更为重要。
HttpAsyncClient特性
- 基于标准的纯Java,HTTP版本1.0和1.1的实现
- 在可扩展的OO框架中完全实现所有HTTP方法(GET,POST,PUT,DELETE,HEAD,OPTIONS和TRACE)。
- 支持使用HTTPS(基于SSL的HTTP)协议进行加密。
- 通过HTTP代理的透明连接。\
- 通过CONNECT方法通过HTTP代理建立的HTTPS连接隧道。
- 连接管理支持并发请求执行。支持设置最大总连接数以及每个主机的最大连接数。检测并关闭过期的连接。
- 在HTTP / 1.0中使用KeepAlive的持久连接和在HTTP / 1.1中的持久性
- 设置连接超时的能力。
- 源代码可根据Apache许可免费获得。
- 基本,摘要,NTLMv1,NTLMv2,NTLM2会话,SNPNEGO和Kerberos身份验证方案。
- 用于自定义身份验证方案的插件机制。
- 自动Cookie处理,用于从服务器读取Set-Cookie:标头,并在适当时在Cookie:标头中发回。
- 自定义Cookie策略的插件机制。
- 支持HTTP / 1.1响应缓存。
- 支持流水线的请求执行和处理。
前言
超文本传输协议(HTTP)可能是当今Internet上使用的最重要的协议。 Web服务,支持网络的设备和网络计算的增长继续将HTTP协议的作用扩展到用户驱动的Web浏览器之外,同时增加了需要HTTP支持的应用程序的数量。
尽管java.net软件包提供了用于通过HTTP访问资源的基本功能,但它并未提供许多应用程序所需的全部灵活性或功能。 HttpClient试图填补这一点通过提供高效,最新且功能丰富的软件包来实现最新的HTTP标准和建议的客户端,从而使此方法无效
HttpClient是为扩展而设计的,同时提供对基本HTTP协议的强大支持,对于构建HTTP感知的客户端应用程序(例如Web浏览器,Web服务)的任何人来说,HttpClient可能都会感兴趣客户端或利用或扩展HTTP协议进行分布式通信的系统。
-
HttpClient作用域
-
基于HttpCore的客户端HTTP传输库
-
基于经典(阻塞)I / O
-
内容不可知
-
HttpClient不是什么
HttpClient不是浏览器。 它是客户端HTTP传输库。 HttpClient的目的是传输和接收HTTP消息,HttpClient将不会尝试处理内容,执行HTML页面中嵌入的javascript,尝试猜测内容类型(如果未显式设置),重新格式化请求/重写位置URI或其他与HTTP传输无关的功能。
最新Pom引入
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpasyncclient</artifactId>
<version>4.1.4</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.10</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.5.10</version>
</dependency>
HttpClient连接池简单封装
支持https 无须认证请求
import com.alibaba.fastjson.JSONObject;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.config.SocketConfig;
import org.apache.http.conn.DnsResolver;
import org.apache.http.conn.HttpConnectionFactory;
import org.apache.http.conn.ManagedHttpClientConnection;
import org.apache.http.conn.routing.HttpRoute;
import org.apache.http.conn.socket.ConnectionSocketFactory;
import org.apache.http.conn.socket.PlainConnectionSocketFactory;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.entity.ByteArrayEntity;
import org.apache.http.entity.ContentType;
import org.apache.http.impl.DefaultConnectionReuseStrategy;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultConnectionKeepAliveStrategy;
import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.DefaultHttpResponseParserFactory;
import org.apache.http.impl.conn.ManagedHttpClientConnectionFactory;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.impl.conn.SystemDefaultDnsResolver;
import org.apache.http.impl.io.DefaultHttpRequestWriterFactory;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.net.ssl.SSLContext;
import java.io.IOException;
import java.io.Serializable;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.concurrent.TimeUnit;
/**
* @author Created by niugang on 2020/3/25/15:13
* HttpClient实现应该是线程安全的。 建议将此类的相同实例重用于多个请求执行。
*/
public class HttpClientV2Utils {
private static final Logger LOGGER = LoggerFactory.getLogger(HttpClientV2Utils.class);
public static CloseableHttpClient httpClient = null;
/**
* 获取HttpClient工具类
*
* @return CloseableHttpClient
* @throws Exception Exception
*/
public static synchronized CloseableHttpClient getHttpClient() throws Exception {
if (httpClient == null) {
//使用 loadTrustMaterial() 方法实现一个信任策略,信任所有证书
SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {
// 信任所有
@Override
public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {
return true;
}
}).build();
// 该主机名验证程序实际上关闭了主机名验证。 它接受任何有效的SSL会话并匹配目标主机
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext, NoopHostnameVerifier.INSTANCE);
//为支持的协议方案创建自定义连接套接字工厂的注册表。
Registry<ConnectionSocketFactory> socketFactoryRegistry = RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", PlainConnectionSocketFactory.INSTANCE)
.register("https", sslsf)
.build();
//HTTPConnection工厂 :配置请求/解析响应
HttpConnectionFactory<HttpRoute, ManagedHttpClientConnection> connFactory = new ManagedHttpClientConnectionFactory(
DefaultHttpRequestWriterFactory.INSTANCE, DefaultHttpResponseParserFactory.INSTANCE);
//DNS解析器
DnsResolver dnsResolver = SystemDefaultDnsResolver.INSTANCE;
//创建池化管理器
PoolingHttpClientConnectionManager manager = new PoolingHttpClientConnectionManager(
socketFactoryRegistry, connFactory, dnsResolver);
// 默认为Socket配置
SocketConfig socketConfig = SocketConfig.custom()
.setTcpNoDelay(true)
.build();
manager.setDefaultSocketConfig(socketConfig);
// 配置永久连接的最大总数或每个路由限制
// 可以保留在池中或由连接管理器租用。
//每个路由的默认最大连接,每个路由实际最大连接为默认为DefaultMaxPreRoute控制,而MaxTotal是控制整个池子最大数
manager.setMaxTotal(100);
manager.setDefaultMaxPerRoute(10);
// 在从连接池获取连接时,连接不活跃多长时间后需要进行一次验证,默认为2s
manager.setValidateAfterInactivity(5 * 1000);
//默认请求配置
//设置连接超时时间 4s
//设置等待数据超时时间,5s
//设置从连接池获取连接的等待超时时间
RequestConfig defaultRequestConfig = RequestConfig.custom()
.setConnectTimeout(4 * 1000)
.setSocketTimeout(5 * 1000)
.setConnectionRequestTimeout(2000)
.build();
httpClient = HttpClients.custom()
.setConnectionManager(manager)
.setDefaultRequestConfig(defaultRequestConfig)
//连接池不是共享模式
.setConnectionManagerShared(false)
.evictIdleConnections(60, TimeUnit.SECONDS)
//定期回收空闲连接
.evictExpiredConnections()
.setConnectionTimeToLive(60, TimeUnit.SECONDS)
//连接重用策略,即是否能keeplive
.setConnectionReuseStrategy(DefaultConnectionReuseStrategy.INSTANCE)
//长连接配置,即获取长连接生产多长时间
.setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE)
//设置重试次数,默认为3次;当前禁用掉(根据需要重启)
.setRetryHandler(new DefaultHttpRequestRetryHandler(0, false))
.build();
// jvm 停止或重启时,关闭连接池释放连接资源(跟数据库连接池类似)
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
try {
httpClient.close();
} catch (IOException e) {
LOGGER.error("HttpClient close exception", e);
}
}
});
}
return httpClient;
}
/**
* get方法封装
* <p>
* 要使用 EntityUtils.consume(response.getEntity());或者EntityUtils.toString(response.getEntity());
* 消费响应,不推荐HttpEntity.getContent.close方法来释放连接,处理不好异常将导致连接不释放,也不推荐CloseableHttpClient#close关闭连接,它将直接关闭
* socket,导致长连接不能复用
*
* @param httpGet httpGet
* @return String
*/
private static String doGet(HttpGet httpGet) {
HttpResponse response = null;
try {
response = getHttpClient().execute(httpGet);
if (response.getStatusLine().getStatusCode() != HttpStatus.SC_OK) {
if (response.getEntity() != null) {
EntityUtils.consume(response.getEntity());
}
LOGGER.info("GET execute exception [{}]->[{}]", response.getStatusLine(), response.getStatusLine().getReasonPhrase());
return "error";
} else {
return EntityUtils.toString(response.getEntity());
}
} catch (Exception e) {
if (response != null) {
try {
EntityUtils.consume(response.getEntity());
} catch (IOException e1) {
LOGGER.info("GET consume response entity exception", e);
}
}
}
return "error";
}
public static void main(String[] args) throws Exception {
HttpGet httpsget = new HttpGet("https://11.12.115.104:8443/bs/gateway/get.do");
httpsget.setHeader("token", "c50e18bc01a62d395a9755b523bc2e50");
String s = doGet(httpsget);
LOGGER.info("doGet :{}", s);
//####1.HTTPS get请求验证 方式二
URI uri = new URIBuilder()
.setScheme("https")
.setHost("11.12.115.104")
.setPort(8443)
.setPath("/bs/gateway/get.do")
.build();
HttpGet httpsget1 = new HttpGet(uri);
httpsget1.setHeader("token", "12fd6749989c7aa60ef34c8d53c0cf71");
// 自定义响应处理器
ResponseHandler<String> responseHandler = new ResponseHandler<String>() {
@Override
public String handleResponse(
final HttpResponse response) throws IOException {
int status = response.getStatusLine().getStatusCode();
if (status >= 200 && status < 300) {
HttpEntity entity = response.getEntity();
return entity != null ? EntityUtils.toString(entity, StandardCharsets.UTF_8) : null;
} else {
throw new ClientProtocolException("Unexpected response status: " + status + "->" + response.getStatusLine().getReasonPhrase());
}
}
};
String execute = getHttpClient().execute(httpsget1, responseHandler);
LOGGER.info("https get excute result {}", execute);
//####2.Https Post请求验证 请求体为json数据
JSONObject jsonObject = new JSONObject();
jsonObject.put("user", "44444");
jsonObject.put("ip", "192.168.1.3");
jsonObject.put("status", "0");
ByteArrayEntity byteArrayEntity = new ByteArrayEntity(jsonObject.toJSONString().getBytes(), ContentType.create("application/json", "UTF-8"));
HttpPost httpspost = new HttpPost("https://11.12.115.104:8443/accessControl/saveDevice.do");
httpspost.setHeader("token", "12fd6749989c7aa60ef34c8d53c0cf71");
httpspost.setEntity(byteArrayEntity);
String httpsGostExecute = getHttpClient().execute(httpspost, responseHandler);
LOGGER.info("https post excute result {}", httpsGostExecute);
//####3.http get
HttpGet httpGet = new HttpGet("http://www.baidu.com");
String httpGetExecute = getHttpClient().execute(httpGet, responseHandler);
LOGGER.info("http get excute result {}", httpGetExecute);
}
}
HttpAsyncClient连接池简单封装
package com.example.demo.utils;
import lombok.SneakyThrows;
import org.apache.http.*;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.config.CookieSpecs;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.concurrent.FutureCallback;
import org.apache.http.config.ConnectionConfig;
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.conn.DnsResolver;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.impl.conn.SystemDefaultDnsResolver;
import org.apache.http.impl.nio.client.CloseableHttpAsyncClient;
import org.apache.http.impl.nio.client.HttpAsyncClients;
import org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFactory;
import org.apache.http.impl.nio.codecs.DefaultHttpResponseParserFactory;
import org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory;
import org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager;
import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
import org.apache.http.impl.nio.reactor.IOReactorConfig;
import org.apache.http.nio.conn.ManagedNHttpClientConnection;
import org.apache.http.nio.conn.NHttpConnectionFactory;
import org.apache.http.nio.conn.NoopIOSessionStrategy;
import org.apache.http.nio.conn.SchemeIOSessionStrategy;
import org.apache.http.nio.conn.ssl.SSLIOSessionStrategy;
import org.apache.http.nio.reactor.ConnectingIOReactor;
import org.apache.http.nio.reactor.IOReactorException;
import org.apache.http.nio.util.HeapByteBufferAllocator;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.net.ssl.SSLContext;
import java.io.IOException;
import java.nio.charset.CodingErrorAction;
import java.nio.charset.StandardCharsets;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.concurrent.Future;
/**
* 此示例演示如何自定义和配置HTTP请求执行和连接管理的最常见方面。
*
* @author Created by niugang on 2020/3/24/11:25
*/
public class HttpAsyncClientUtils {
private static final Logger LOGGER = LoggerFactory.getLogger(HttpAsyncClientUtils.class);
public static CloseableHttpAsyncClient httpclient = null;
/**
* 获取 ttpAsyncClient
*
* @return CloseableHttpAsyncClient
* @throws KeyStoreException KeyStoreException
* @throws NoSuchAlgorithmException NoSuchAlgorithmException
* @throws IOReactorException IOReactorException
*/
public static synchronized CloseableHttpAsyncClient getHttpAsyncClient() throws KeyStoreException, NoSuchAlgorithmException, KeyManagementException, IOReactorException {
if (httpclient == null) {
// HTTPConnection工厂 :配置请求/解析响应
NHttpConnectionFactory<ManagedNHttpClientConnection> connFactory = new ManagedNHttpClientConnectionFactory(
DefaultHttpRequestWriterFactory.INSTANCE, DefaultHttpResponseParserFactory.INSTANCE, HeapByteBufferAllocator.INSTANCE);
//ssl 连接设置 无须证书也能访问 https
//使用 loadTrustMaterial() 方法实现一个信任策略,信任所有证书
SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(null, new TrustStrategy() {
// 信任所有
@Override
public boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {
return true;
}
}).build();
// 为支持的协议方案创建自定义连接套接字工厂的注册表。
Registry<SchemeIOSessionStrategy> sessionStrategyRegistry = RegistryBuilder.<SchemeIOSessionStrategy>create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", new SSLIOSessionStrategy(sslContext, NoopHostnameVerifier.INSTANCE))
.build();
//DNS解析器
DnsResolver dnsResolver = SystemDefaultDnsResolver.INSTANCE;
// Create I/O reactor configuration
IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
.setIoThreadCount(Runtime.getRuntime().availableProcessors())
.setConnectTimeout(30000)
.setSoTimeout(30000)
.build();
// 创建一个定制的I/O reactort
ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor(ioReactorConfig);
// 使用自定义配置创建连接管理器。
PoolingNHttpClientConnectionManager connManager = new PoolingNHttpClientConnectionManager(
ioReactor, connFactory, sessionStrategyRegistry, dnsResolver);
//创建连接配置
ConnectionConfig connectionConfig = ConnectionConfig.custom()
.setMalformedInputAction(CodingErrorAction.IGNORE)
.setUnmappableInputAction(CodingErrorAction.IGNORE)
.setCharset(Consts.UTF_8)
.build();
// 将连接管理器配置为默认使用或针对特定主机使用连接配置。
connManager.setDefaultConnectionConfig(connectionConfig);
// 配置永久连接的最大总数或每个路由限制
// 可以保留在池中或由连接管理器租用。
//每个路由的默认最大连接,每个路由实际最大连接为默认为DefaultMaxPreRoute控制,而MaxTotal是控制整个池子最大数
connManager.setMaxTotal(100);
connManager.setDefaultMaxPerRoute(10);
// 创建全局请求配置
RequestConfig defaultRequestConfig = RequestConfig.custom()
.setCookieSpec(CookieSpecs.DEFAULT)
.setSocketTimeout(5 * 1000)
.setConnectTimeout(5 * 1000)
.setExpectContinueEnabled(true)
.build();
// Create an HttpClientUtils with the given custom dependencies and configuration.
httpclient = HttpAsyncClients.custom()
.setConnectionManager(connManager)
.setDefaultRequestConfig(defaultRequestConfig)
.build();
//jvm 停止或重启时,关闭连接池释放连接资源(跟数据库连接池类似)
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
try {
httpclient.close();
} catch (IOException e) {
LOGGER.error("HttpClient close exception", e);
}
}
});
}
return httpclient;
}
public static HttpResponse doGet(String url) {
CloseableHttpAsyncClient httpAsyncClient = null;
try {
final HttpGet httpget = new HttpGet(url);
LOGGER.info("Executing request {}", httpget.getRequestLine());
httpAsyncClient = getHttpAsyncClient();
httpAsyncClient.start();
// Pass local context as a parameter
Future<HttpResponse> future = httpAsyncClient.execute(httpget, new FutureCallback<HttpResponse>() {
@SneakyThrows
@Override
public void completed(final HttpResponse response2) {
LOGGER.info("{} -> {} completed", httpget.getRequestLine(), response2.getStatusLine());
}
@Override
public void failed(final Exception ex) {
LOGGER.error("{}->", httpget.getRequestLine(), ex);
}
@Override
public void cancelled() {
LOGGER.error("{} cancelled", httpget.getRequestLine());
}
});
//等待响应结果
return future.get();
} catch (Exception e) {
LOGGER.error("GET execute exception ", e);
} finally {
if (httpAsyncClient != null) {
//直接关闭socket,会导致长连接不能复用,所以返回Future,调用放调用get方法,将一直处于阻塞状态
try {
httpAsyncClient.close();
} catch (IOException e) {
LOGGER.error("GET execute httpAsyncClient close exception ", e);
}
}
}
return null;
}
public static void main(String[] args) throws Exception {
HttpResponse response = doGet("https://11.12.115.104:8443/serverStatus/serverInfo");
if (response == null) {
return;
}
LOGGER.info("Response: {}", response.getStatusLine());
int status = response.getStatusLine().getStatusCode();
if (status >= 200 && status < 300) {
HttpEntity entity = response.getEntity();
String res = entity != null ? EntityUtils.toString(entity, StandardCharsets.UTF_8) : null;
LOGGER.info("result->{}", res);
} else {
throw new ClientProtocolException("Unexpected response status: " + status + "->" + response.getStatusLine().getReasonPhrase());
}
}
}
微信公众号