SpringBoot集成ElasticSearch
1、使用方法
springboot里有两个比较好用的操作ES的工具，
一个是Jest（默认不生效），连接的是es的9200端口（Http Restful）。
需要导入jest的工具包，(io.searchbox.client.JestClient)
一个是spring data elasticsearch，前者连接的是es的9300端口（TCP连接，把自己伪装成ES的一个节点）。
- Client节点信息clusterNodes；clusterName
2）ElasticSearchTemplate操作es
3）编写一个ElasticSearchRepository的子接口来操作ES。
2、Jest使用方式
pom.xml
<!--SpringBoot默认使用SpringData ElasticSearch模块进行操作-->
<!--<dependency>-->
<!--<groupId>org.springframework.boot</groupId>-->
<!--<artifactId>spring-boot-starter-data-elasticsearch</artifactId>-->
<!--</dependency>-->
<!-- https://mvnrepository.com/artifact/io.searchbox/jest -->
<dependency>
<groupId>io.searchbox</groupId>
<artifactId>jest</artifactId>
<version>6.3.1</version>
</dependency>
application.properties
spring.elasticsearch.jest.uris=http://192.168.8.101:9200
Bean
public class Article {
@JestId
private Integer id;
private String autor;
private String title;
private String content;
public Article() {
}
public Article(Integer id, String autor, String title, String content) {
this.id = id;
this.autor = autor;
this.title = title;
this.content = content;
}
/*
@Getter
@Setter
*/
@Test
@SpringBootTest
class ElaticsearchApplicationTests {
@Autowired
JestClient jestClient;
@Test
void contextLoads() {
}
@Test
public void jest_insert() {
// 给Es中索引（保存）一个文档
Article article = new Article();
article.setId(1);
article.setTitle("Effect Java");
article.setAutor("Joshua Bloch");
article.setContent("Hello World");
// 构建一个索引功能
Index index = new Index.Builder(article).index("demo_index").type("article").build();
try {
//执行
jestClient.execute(index);
} catch (IOException e) {
e.printStackTrace();
}
}
//查询数据
@Test
public void jest_search(){
// 查询表达式
String json = "{\n" +
" \"query\" : {\n" +
" \"match\" : {\n" +
" \"content\" : \"Hello\"\n" +
" }\n" +
" }\n" +
"}";
// 构建搜索操作
Search search = new Search.Builder(json).addIndex("demo_index").addType("article").build();
// 执行
try {
SearchResult result = jestClient.execute(search);
System.out.println(result.getJsonString());
} catch (IOException e) {
e.printStackTrace();
}
}
}
3、SpringData ElasticSearch使用方式
Elasticsearch(ES)有两种连接方式：transport、rest。
通过监听9300端口tcp链接进行数据传输，他可以触摸到es的API和结构。
在Springboot集成ES的两种方式种，一般有spring-boot-starter-data-elasticsearch和Spring-data-elasticsearch。
其中spring-boot-starter-data-elasticsearch。第一个是Springboot官方的整合包，使用更方便。
但是更新缓慢，支持版本较低。而ES版本更新较快。版本不一致直接整合不上。
而Spring-data-elasticsearch对版本支持稍微好一点。版本对应关系。你会发现对新版本支持还是比较差的。
transport通过TCP方式访问ES(只支持java),rest方式通过http API 访问ES(没有语言限制)。
ES官方建议使用rest方式, transport 在7.0版本中不建议使用，在8.X的版本中废弃。
请注意，SpringBoot是2.2.0.RELEASE才兼容elasticsearch 7.x(http方式）
具体使用方法
1）集成ElasticsearchRepository接口
pom.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.1.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
......
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
application.properties
spring.elasticsearch.rest.uris=http://192.168.8.101:9200
repository.BookRepository
public interface BookRepository extends ElasticsearchRepository<Book,Integer>{
}
ElasticsearchRepository
@NoRepositoryBean
public interface ElasticsearchRepository<T, ID> extends ElasticsearchCrudRepository<T, ID> {
<S extends T> S index(S var1);
<S extends T> S indexWithoutRefresh(S var1);
Iterable<T> search(QueryBuilder var1);
Page<T> search(QueryBuilder var1, Pageable var2);
Page<T> search(SearchQuery var1);
Page<T> searchSimilar(T var1, String[] var2, Pageable var3);
void refresh();
Class<T> getEntityClass();
}
注意ES7版本以后：
Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})."]
[POST http://192.168.8.101:9200/springdataes_index/book?timeout=1m] returned 1 warnings: [299 Elasticsearch-7.4.1-fc0eeb6e2c25915d63d871d344e3d0b45ea0ea1e "[types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id})."]
@Autowired
BookRepository bookRepository;
......
@Test
public void springdataes_search(){
Book book = new Book();
book.setId(1);
book.setBookName("BookName");
book.setAuthor("Author");
bookRepository.index(book);
}
BookRepository增加自定义方法
public interface BookRepository extends ElasticsearchRepository<Book,Integer>{
/*
除了继承，支持自定义方法
*/
public List<Book> findByBookNameLike(String bookName);
}
执行下面的测试方法会报错，说明对7.4.1 高版本的ES支持还是不好。这里暂时不介绍更多。
List<Book> books= bookRepository.findByBookNameLike("BookName");
for (Book book:books) {
System.out.println(book);
}
java.lang.NoSuchMethodError: org.elasticsearch.search.SearchHits.getTotalHits()J
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doCount(ElasticsearchRestTemplate.java:604)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.count(ElasticsearchRestTemplate.java:566)
at org.springframework.data.elasticsearch.repository.query.ElasticsearchPartQuery.execute(ElasticsearchPartQuery.java:81)
......
4、elasticsearch-rest-high-level-client
pom.xml
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>${elasticsearch.version}</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>${elasticsearch.version}</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>${elasticsearch.version}</version>
</dependency>
rest,不难想到http，restclient就是采用http进行交互。
restclient相比transport最大的好处就是—对于版本兼容性较好。
然而，restclient也分为两种——high—level和low—level两种，两者原理基本一致，区别最大的就是封装性。
low—level各种操作都要你自己封装，并且java本身不支持json还需要引用第三方包。
而high—level是针对elasticsearch的api进行高级封装，和elasticsearch的版本关联大一些。
low—level就行原生爬虫，啥东西都要你自己写，而high—level就像是框架一般，各种方法帮你稍微封装好。使用起来较为方便。
官方明确说明transportclient在elasticsearch高版本会直接遗弃。只支持restclient。
为了顺应ES的潮流，还是要用restclient。并且transportclient在高并发会有性能问题。
ESConfig
@Configuration
public class ESConfig {
private static String hosts = "192.168.8.101"; // 集群地址，多个用,隔开
private static int port = 9200; // 使用的端口号
private static String schema = "http"; // 使用的协议
private static ArrayList hostList = null;
private static int connectTimeOut = 1000; // 连接超时时间
private static int socketTimeOut = 30000; // 连接超时时间
private static int connectionRequestTimeOut = 500; // 获取连接的超时时间
private static int maxConnectNum = 100; // 最大连接数
private static int maxConnectPerRoute = 100; // 最大路由连接数
static {
hostList = new ArrayList<>();
String[] hostStrs = hosts.split(",");
for (String host : hostStrs) {
hostList.add(new HttpHost(host, port, schema));
}
}
@Bean
public RestHighLevelClient client() {
RestClientBuilder builder = RestClient.builder((HttpHost[]) hostList.toArray(new HttpHost[0]));
// 异步httpclient连接延时配置
builder.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
@Override
public Builder customizeRequestConfig(Builder requestConfigBuilder) {
requestConfigBuilder.setConnectTimeout(connectTimeOut);
requestConfigBuilder.setSocketTimeout(socketTimeOut);
requestConfigBuilder.setConnectionRequestTimeout(connectionRequestTimeOut);
return requestConfigBuilder;
}
});
// 异步httpclient连接数配置
builder.setHttpClientConfigCallback(new HttpClientConfigCallback() {
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
httpClientBuilder.setMaxConnTotal(maxConnectNum);
httpClientBuilder.setMaxConnPerRoute(maxConnectPerRoute);
return httpClientBuilder;
}
});
RestHighLevelClient client = new RestHighLevelClient(builder);
return client;
}
}
Bean
@Document(indexName = "springdataes_index"/*,type = "book"*/)
public class Book {
private Integer id;
private String bookName;
private String author;
public Book() {
}
public Book(Integer id, String bookName, String author) {
this.id = id;
this.bookName = bookName;
this.author = author;
}
public Integer getId() {
return id;
}
public void setId(Integer id) {
this.id = id;
}
public String getBookName() {
return bookName;
}
public void setBookName(String bookName) {
this.bookName = bookName;
}
public String getAuthor() {
return author;
}
public void setAuthor(String author) {
this.author = author;
}
@Override
public String toString() {
return "Book{" +
"id=" + id +
", bookName='" + bookName + '\'' +
", author='" + author + '\'' +
'}';
}
}
@Test
/*
第三种elastic elasticsearch-rest-high-level-client
*/
@Autowired
private RestHighLevelClient client;
......
@Test
public void highLevelClient(){
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
// sourceBuilder.from(0);
// sourceBuilder.size(10);
sourceBuilder.fetchSource(new String[]{"id","author","bookName"}, new String[]{});
MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("author", "Author");
// TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("tag", "体育");
// RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("publishTime");
// rangeQueryBuilder.gte("2018-01-26T08:00:00Z");
// rangeQueryBuilder.lte("2018-01-26T20:00:00Z");
BoolQueryBuilder boolBuilder = QueryBuilders.boolQuery();
boolBuilder.must(matchQueryBuilder);
// boolBuilder.must(termQueryBuilder);
// boolBuilder.must(rangeQueryBuilder);
sourceBuilder.query(boolBuilder);
SearchRequest searchRequest = new SearchRequest("springdataes_index");
// searchRequest.types(type);
searchRequest.source(sourceBuilder);
try {
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
SearchHits hits = response.getHits();
SearchHit[] searchHits = hits.getHits();
for (SearchHit hit : searchHits) {
System.out.println("search -> " + hit.getSourceAsString());
/*
search -> {"author":"Author","id":2,"bookName":"BookName"}
search -> {"author":"Author","id":1,"bookName":"BookName"}
*/
JSONObject bookJson = JSONObject.parseObject(hit.getSourceAsString());
Book book = JSON.toJavaObject(bookJson,Book.class);
System.out.println(book);
}
// System.out.println(JSON.toJSONString(response));
/*
{
"took": 106,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 0.2876821,
"hits": [{
"_index": "springdataes_index",
"_type": "book",
"_id": "2",
"_score": 0.2876821,
"_source": {
"author": "Author"
}
}, {
"_index": "springdataes_index",
"_type": "book",
"_id": "1",
"_score": 0.2876821,
"_source": {
"author": "Author"
}
}]
}
}
*/
} catch (IOException e) {
e.printStackTrace();
}
}
多条件查询
@Test
public void highLevelClientDemo(){
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.fetchSource(new String[]{"id","author","bookName"}, new String[]{});
MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("author", "Author");
MatchQueryBuilder matchQueryBuilder1 = QueryBuilders.matchQuery("id", "2");
BoolQueryBuilder boolBuilder = QueryBuilders.boolQuery();
/******************************************************************************************/
boolBuilder.must(matchQueryBuilder).must(matchQueryBuilder1);
/******************************************************************************************/
sourceBuilder.query(boolBuilder);
SearchRequest searchRequest = new SearchRequest("springdataes_index");
searchRequest.source(sourceBuilder);
try {
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
SearchHits hits = response.getHits();
SearchHit[] searchHits = hits.getHits();
for (SearchHit hit : searchHits) {
System.out.println("search -> " + hit.getSourceAsString());
/*
search -> {"author":"Author","id":2,"bookName":"BookName"}
*/
JSONObject bookJson = JSONObject.parseObject(hit.getSourceAsString());
Book book = JSON.toJavaObject(bookJson,Book.class);
System.out.println(book);
}
} catch (IOException e) {
e.printStackTrace();
}
}
ELK日志搜索的具体例子
@Test
public void highLevelClientDemo1(){
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
sourceBuilder.fetchSource(new String[]{"log_level","log_thread","log_date","log_class","log_content"}, new String[]{});
MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("log_content", "xx1");
BoolQueryBuilder boolBuilder = QueryBuilders.boolQuery();
boolBuilder.must(matchQueryBuilder);
sourceBuilder.query(boolBuilder);
SearchRequest searchRequest = new SearchRequest("log-2019.11.24");
searchRequest.source(sourceBuilder);
try {
SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
SearchHits hits = response.getHits();
SearchHit[] searchHits = hits.getHits();
for (SearchHit hit : searchHits) {
System.out.println("search -> " + hit.getSourceAsString());
/*上节基于ELK的日志搜索
search -> {"log_thread":"restartedMain","log_date":"2019-03-21 09:20:01,104","log_level":"INFO","log_class":"com.xxx.xxx.XXXApplication","log_content":"xx1"}
*/
}
} catch (IOException e) {
e.printStackTrace();
}
}
更新id为1的 Book对象的author字段"Author"-》UpdatedAuthor
/*
更新id为1的 Book对象的author字段"Author"-》UpdatedAuthor
*/
@Test
public void update() throws IOException {
String index = "springdataes_index";
String type = "";
Book update_book = new Book();
update_book.setId(1);
update_book.setAuthor("UpdatedAuthor");
UpdateRequest request = new UpdateRequest(index, update_book.getId().toString());
request.doc(JSON.toJSONString(update_book), XContentType.JSON);
UpdateResponse updateResponse = client.update(request, RequestOptions.DEFAULT);
System.out.println("update: " + JSON.toJSONString(updateResponse));
}
