ElasticSearch
1.es与数据库对应关系
index —— 数据库
document —— 一条数据
field —— 一个字段
2.操作
基本命令补充:
GET /_cat/nodes 查看所有节点
GET /_cat/health 查看es健康状况
GET /_cat/master 查看主节点
GET /_cat/indices 查看所有索引
2.1.index相关操作
增加索引:url:http://ip地址:9200/索引名称
,使用PUT方法
返回结果:
如果想要获取index信息,则请求路径不用变,只需要将请求方法替换为get即可
如果要删除索引,则使用DELETE方式
查看所有的索引:http://IP地址:9200/_cat/indices
使用get方式,返回结果如下:
如果希望详细展示,则在后面加上?v
,即http://IP地址:9200/_cat/indices?v
2.2.document相关操作
创建文档,添加数据:使用POST方式,url:http://IP地址:9200/索引名称/_doc
,同时要设置json格式的请求体
如果希望在增加数据时,自定义ID,即可在_doc
后面增加ID值,即http://IP地址:9200/索引名称/_doc/ID值
如果想要查询指定文档,则使用get方式,url使用http://IP地址:9200/索引名称/_doc/ID值
查询指定index下的所有文档:使用get方式,url使用http://IP地址:9200/索引名称/_search
对已经建立的文档进行完全覆盖:由于每次都会完全修改文档,因此是幂等性的,使用PUT方式,url与查询时相同,使用http://IP地址:9200/索引名称/_doc/ID值
,但是同时要写明json格式的请求体
对已经建立的文档进行部分覆盖:此种方式是非幂等性的,因此只能使用POST方式,url使用http://IP地址:9200/索引名称/_update/ID值
,body则使用如下的json格式:
{
"doc": {
"属性名": "属性值"
}
}
删除指定文档:使用DELETE方式,url使用http://IP地址:9200/索引名称/_doc/ID值
2.3.条件查询
使用GET/POST方式,url使用http://IP地址:9200/索引名称/_search?q=属性名:属性值
或者使用get方式,并在请求体中使用json:
{
"query":{
"match": {
"属性名":"属性值"
}
}
}
全量查询:使用GET/POST方式,url使用http://IP地址:9200/索引名称/_search
,请求体json为:
{
"query":{
"match_all":{
}
}
}
分页查询:使用GET/POST方式,url使用http://IP地址:9200/索引名称/_search
,请求体json为:(表示从第一条开始,查询两条数据)
{
"query":{
"match_all":{
}
},
"from": 0,
"size": 2
}
若希望结果仅显示指定字段,请求体json可以按照如下方式(使用_source)进行编写:
{
"query":{
"match_all":{
}
},
"from": 2,
"size": 2,
"_source":["属性名"]
}
如果希望进行排序,则可以在请求体中加入sort
:
{
"query":{
"match_all":{
}
},
"from": 2,
"size": 2,
"_source":["属性1","属性2"],
"sort": {
"属性名":{
"order": "desc"
}
}
}
2.4.多条件查询
使用GET/POST方式,url使用http://IP地址:9200/索引名称/_search
2.4.1.多个条件同时成立
{
"query":{
"bool":{
"must": [
{
"match":{
"属性名1": "属性值1"
}
},
{
"match":{
"属性名2": "属性值2"
}
}
]
}
}
}
2.4.2.多个条件有一个成立
{
"query":{
"bool":{
"should": [
{
"match":{
"属性名1": "属性值1"
}
},
{
"match":{
"属性名2": "属性值2"
}
}
]
}
}
}
2.5.范围查询(使用filter)
使用GET/POST方式,url使用http://IP地址:9200/索引名称/_search
请求体json:
{
"query":{
"bool":{
"should": [
{
"match":{
"category": "xiaomi"
}
},
{
"match":{
"price": 3999
}
}
],
"filter":{
"range": {
"属性名": {
"gt": 指定值 // 表示属性值大于指定值的数据
}
}
}
}
}
}
2.6.匹配注意
注意:
{
"query":{
"match":{
"category":"小华"
}
}
}
此时查询会将小华
分解为小
、华
进行匹配,如果希望完全匹配,不进行分解,则可以使用match_phrase
:
{
"query":{
"match_phrase":{
"category":"小华"
}
}
}
2.7.高亮显示
{
"query":{
"match":{
"category":"小"
}
},
"highlight":{
"fields":{
"属性名":{
}
}
}
}
返回结果:
2.8.分组
{
"aggs":{ // 聚合操作
"price_group":{ // 名称,随意起
"terms": { // 分组
"field": "属性名" // 分组字段
}
}
},
"size": 0 // 不显示原始数据
}
2.9.求平均值
{
"aggs":{ // 聚合操作
"price_avg":{ // 名称,随意起
"avg": { // 求平均值
"field": "属性名" // 分组字段
}
}
},
"size": 0 // 不显示原始数据
}
2.10.建立映射(类比于mysql表结构)
使用PUT方式,url为http://192.168.26.128:9200/index名称/_mapping
{
"properties": {
"name": {
"type": "text", // 可以分词
"index": true
},
"sex":{
"type":"keyword", // 不能分词,必须完整匹配
"index":true
},
"phone":{
"type":"keyword",
"index":false
}
}
}
改为GET请求,可以查询映射信息
3.程序操作
3.1.导入依赖:
<dependencies>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>7.8.0</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>7.8.0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.9</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
</dependencies>
在resources下增加log4j.properties:
#定义输出级别
log4j.rootLogger=DEBUG,Console,File
#日志输出方式:控制台输出
log4j.appender.Console=org.apache.log4j.ConsoleAppender
log4j.appender.Console.Target=System.out
log4j.appender.Console.Encoding=GBK
#可以灵活地指定布局模式
log4j.appender.Console.layout=org.apache.log4j.PatternLayout
#log4j.appender.Console.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss.SSS} -%p (%F\:%L)- %m%n
#打印格式:2017-08-11 15:36 -DEBUG (HttpServletBean.java:174)- Servlet 'mvc' configured successfully
log4j.appender.Console.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm} -%p (%F\:%L)- %m%n
3.2.建立连接
public class ESClient {
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
restHighLevelClient.close();
}
}
3.3.索引操作
@Test
public void test() throws IOException {
Logger logger = LoggerFactory.getLogger(EsIndexCreate.class);
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
// 创建索引
CreateIndexResponse response = restHighLevelClient.indices().create(new CreateIndexRequest("user"), RequestOptions.DEFAULT);
// 响应状态
logger.debug(String.valueOf(response.isAcknowledged()));
// 查询索引
GetIndexResponse resp = restHighLevelClient.indices().get(new GetIndexRequest("user"), RequestOptions.DEFAULT);
System.out.println("----------------------------------");
System.out.println(resp.getAliases());
System.out.println(resp.getMappings());
System.out.println(resp.getSettings());
System.out.println("----------------------------------");
// 删除索引
AcknowledgedResponse deleteResp = restHighLevelClient.indices().delete(new DeleteIndexRequest("user"), RequestOptions.DEFAULT);
logger.debug(String.valueOf(deleteResp.isAcknowledged()));
restHighLevelClient.close();
}
3.4.数据操作
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
User user = new User();
user.setName("张三");
user.setAge(30);
user.setSex("男");
ObjectMapper mapper = new ObjectMapper();
// 将发送数据转为json
String json = mapper.writeValueAsString(user);
IndexResponse indexResp = restHighLevelClient.index(new IndexRequest("user").id("1001").source(json, XContentType.JSON), RequestOptions.DEFAULT);
System.out.println(indexResp.getResult());
// 局部更新
UpdateRequest updateRequest = new UpdateRequest();
UpdateResponse updateResp = restHighLevelClient.update(updateRequest.index("user").id("1001").doc(XContentType.JSON, "age", 31), RequestOptions.DEFAULT);
System.out.println(updateResp.getResult());
// 查询
GetResponse getResp = restHighLevelClient.get(new GetRequest().index("user").id("1001"), RequestOptions.DEFAULT);
System.out.println(getResp.getSource());
// 删除
restHighLevelClient.delete(new DeleteRequest().index("user").id("1001"), RequestOptions.DEFAULT);
restHighLevelClient.close();
}
3.5.批量数据操作
bulk是将多个操作进行统一打包,进行执行
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
BulkRequest bulkRequest = new BulkRequest();
bulkRequest.add(new IndexRequest().index("user").id("1001").source(XContentType.JSON, "name", "zhangsan", "age", 30, "sex", "男"));
bulkRequest.add(new IndexRequest().index("user").id("1002").source(XContentType.JSON, "name", "lisi", "age", 30, "sex", "女"));
bulkRequest.add(new IndexRequest().index("user").id("1003").source(XContentType.JSON, "name", "wangwu", "age", 40, "sex", "男"));
bulkRequest.add(new IndexRequest().index("user").id("1004").source(XContentType.JSON, "name", "wangwu1", "age", 40, "sex", "女"));
bulkRequest.add(new IndexRequest().index("user").id("1005").source(XContentType.JSON, "name", "wangwu2", "age", 50, "sex", "男"));
bulkRequest.add(new IndexRequest().index("user").id("1006").source(XContentType.JSON, "name", "wangwu3", "age", 50, "sex", "女"));
User user = new User();
user.setName("张三");
user.setAge(30);
user.setSex("男");
bulkRequest.add(new IndexRequest().index("user").id("1001").source(new ObjectMapper().writeValueAsString(user), XContentType.JSON));
bulkRequest.add(new IndexRequest().index("user").id("1002").source(new ObjectMapper().writeValueAsString(user), XContentType.JSON));
bulkRequest.add(new DeleteRequest().index("user").id("1003"));
restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
restHighLevelClient.close();
}
3.6.全量查询
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
BulkRequest bulkRequest = new BulkRequest();
bulkRequest.add(new IndexRequest().index("user").id("1001").source(XContentType.JSON, "name", "zhangsan", "age", 30, "sex", "男"));
bulkRequest.add(new IndexRequest().index("user").id("1002").source(XContentType.JSON, "name", "lisi", "age", 30, "sex", "女"));
bulkRequest.add(new IndexRequest().index("user").id("1003").source(XContentType.JSON, "name", "wangwu", "age", 40, "sex", "男"));
bulkRequest.add(new IndexRequest().index("user").id("1004").source(XContentType.JSON, "name", "wangwu1", "age", 40, "sex", "女"));
bulkRequest.add(new IndexRequest().index("user").id("1005").source(XContentType.JSON, "name", "wangwu2", "age", 50, "sex", "男"));
bulkRequest.add(new IndexRequest().index("user").id("1006").source(XContentType.JSON, "name", "wangwu3", "age", 50, "sex", "女"));
restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("user");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.matchAllQuery());
searchRequest.source(searchSourceBuilder);
// 查询索引中全部数据
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
System.out.println(searchResponse.getHits());
restHighLevelClient.close();
}
3.7.条件查询
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("user");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.termQuery("age", 30));
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
System.out.println(searchResponse.getHits());
restHighLevelClient.close();
}
3.8.分页查询
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("user");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
// 查询前2条数据,并按照年龄升序,并且仅返回指定字段
String[] exludes = {};
String[] includes = {"name"};
searchSourceBuilder.query(QueryBuilders.matchAllQuery()).from(0).size(2).sort("age", SortOrder.ASC).fetchSource(includes, exludes);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
System.out.println("---------------------------------");
for (SearchHit hit : searchResponse.getHits()) {
System.out.println(hit);
}
restHighLevelClient.close();
}
3.9.组合查询
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("user");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.boolQuery().must(QueryBuilders.matchQuery("age", 30)).must(QueryBuilders.matchQuery("sex", "男")));
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
System.out.println("---------------------------------");
for (SearchHit hit : searchResponse.getHits()) {
System.out.println(hit);
}
restHighLevelClient.close();
}
3.10.范围查询
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("user");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.rangeQuery("age").gte(40));
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
System.out.println("---------------------------------");
for (SearchHit hit : searchResponse.getHits()) {
System.out.println(hit);
}
restHighLevelClient.close();
}
3.11.模糊查询
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("user");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
// 模糊匹配,且最多只能有1个字符的差别
searchSourceBuilder.query(QueryBuilders.fuzzyQuery("name", "wangwu").fuzziness(Fuzziness.ONE));
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
System.out.println("---------------------------------");
for (SearchHit hit : searchResponse.getHits()) {
System.out.println(hit);
}
restHighLevelClient.close();
3.12.高亮显示
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("user");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.termsQuery("name", "zhangsan"));
// 设置高亮,同时设置高亮部分被什么标签所包含
searchSourceBuilder.highlighter(new HighlightBuilder().preTags("<span style='color:red'>").postTags("</span>").field("name"));
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
System.out.println("---------------------------------");
for (SearchHit hit : searchResponse.getHits()) {
System.out.println(hit);
}
restHighLevelClient.close();
}
3.13.聚合查询
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("user");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.aggregation(AggregationBuilders.max("maxAge").field("age"));
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
restHighLevelClient.close();
}
3.14.分组查询
@Test
public void test() throws IOException {
RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("192.168.26.128", 9200, "http")));
SearchRequest searchRequest = new SearchRequest();
searchRequest.indices("user");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.aggregation(AggregationBuilders.terms("age_group").field("age"));
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
restHighLevelClient.close();
}
4.集群部署es
单机部署es
集群部署需要修改config下的elasticsearch.yml文件:
# 集群名称
cluster.name: my-cluster
# 节点名称
node.name: node-1001
# ES服务器IP
network.host: 0.0.0.0
# 端口号
http.port: 1001
# 通信端口
transport.tcp.port: 9301
# 是否是master节点
node.master: true
# 是否是数据节点
node.data: true
# 跨域配置
http.cors.enabled: true
http.cors.allow-origin: "*"
cluster.initial_master_nodes: ["node-1001"]
# 增加发现其余es的配置:
discovery.seed_hosts: ["IP:9301"] # 配置其他ES服务器的通信端口
discovery.zen.fd.ping_timeout: 1m
discovery.zen.fd.ping_retries: 5
多个ES需要变更node.name
、http.port
、transport.tcp.port
如果启动过程中出现下面的情况:
针对[1]需要将容器启动配置项中的jvm初始堆内存大小和jvm最大堆内存大小改为一致的,避免jvm堆内存扩容时将堆内存进行锁定,导致全部用户线程暂停
针对[2]需要修改/etc/sysctl.conf
,在最后一行添加vm.max_map_count = 262144
,之后使用sysctl -p
使得配置生效
启动完毕后可以使用GET方式访问http://IP地址:端口号/_cluster/health
查看集群状态
或者使用http://IP:端口号/_cat/nodes
查看集群下的所有节点
5.设置分片及副本
使用PUT方式添加索引,url为http://IP地址:端口号/索引名称
,
请求体为如下的json格式:(表示有3个分片,一份备份【每个分片均有一份备份】)
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1
}
}
可以使用docker安装elasticsearch-head
来查看集群分片和副本情况:
docker pull mobz/elasticsearch-head:5
docker run -d -p 9100:9100 docker.io/mobz/elasticsearch-head:5
通过浏览器访问9100端口即可
分片数量在创建索引时便定死,但是在运行过程中可以改变副本的数量
使用PUT方式,url为http://IP地址:端口号/索引名称/_settings
请求体为如下的json格式:
{
"number_of_replicas": 2
}
6.分析器
6.1.测试
使用GET/POST方法,url为http://IP地址:端口号/analyze
请求体json为
{
"analyzer": "standard",
"text": "Text To Analyze"
}
6.2.安装中文分词器插件
对于中文,推荐使用IK中文分词器:
下载地址:https://github.com/medcl/elasticsearch-analysis-ik/releases
在elasticsearch的plugins目录下创建ik文件夹,将压缩包解压后放入,重新elasticsearch即可
测试url不变,请求体json为
{
"analyzer": "ik_max_word",
"text": "测试单词"
}
ik_max_word会将文本做最细粒度的拆分
ik_smart会将文本做最粗粒度的拆分
6.3.补充自定义词条
部分分词器不识别的词条,可以自定义字典:
在plugins/ik
文件夹下,找到config目录,在其下创建custom.dic
文件,写入词条
同时打开IKAnalyzer.cfg.xml
文件,进行修改:
<!--用户可以在这里配置自己的扩展字典 -->
<entry key="ext_dict">custom.dic</entry>
之后重启ES
7.kibana使用
下载:(注意版本要与ES相同)
docker pull kibana:7.12.0
运行:
docker run --name kibana -v /opt/kibana/config:/usr/share/kibana/config -p 5601:5601 -d kibana:7.12.0
或者修改kibana的配置文件kibana.yml(kibana容器内配置文件地址:/usr/share/kibana/config
)
# 端口号
server.port: 5601
# 允许远程访问地址
server.host: 0.0.0.0
# ES服务器地址
elasticsearch.hosts: ["http://192.168.88.132:9200"]
# 索引名
kibana.index: ".kibana"
# 支持中文
i18n.locale: "zh-CN"
8.SpringData整合ES
引入依赖:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
编写实体类:
@Data
@AllArgsConstructor
@NoArgsConstructor
public class Person {
private String name;
public int age;
public void show() {
System.out.println("show");
}
private Person(String name) {
this.name = name;
}
private String showNation(String nation) {
System.out.println("国籍为" + nation);
return nation;
}
}
编写DAO层:(第二个泛型表示ID类型)
@Repository
public interface ProductDao extends ElasticsearchRepository<Product, Long> {
}
之后注入DAO层实体类,进行操作即可
如分页查询:
@Autowired
private ProductDao productDao;
@Test
void contextLoads() {
Sort sort = Sort.by(Sort.Direction.DESC, "id");
int from = 0;
int size = 5;
PageRequest pageRequest = PageRequest.of(from, size, sort);
Iterable<Product> products = productDao.findAll(pageRequest);
for (Product product : products) {
System.out.println(product);
}
}
本文作者:kanaliya
本文链接:https://www.cnblogs.com/kanaliya/p/15976315.html
版权声明:本作品采用知识共享署名-非商业性使用-禁止演绎 2.5 中国大陆许可协议进行许可。
【推荐】国内首个AI IDE,深度理解中文开发场景,立即下载体验Trae
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· DeepSeek 开源周回顾「GitHub 热点速览」
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布:重大改进与新特性概览!
· AI与.NET技术实操系列(二):开始使用ML.NET
· 单线程的Redis速度为什么快?