elasticsearch-入门(一)
相关文档
不错的中文文档：https://nasuyun.com/docs/api-agg-bucket/api-agg-bucket-filters
es中文社区:https://elasticsearch.cn/question/6899
esapi https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-rest-overview.html
什么是elasticsearch
elasticsearch是java基础apache lucene的开源搜索引擎，开箱即用，elasticsearch有很多默认的合理的配置
elasticsearch作用
全文检索，分布式实时文件存储,分布式实时分析引擎,可以扩展到上百台服务器处理PB级结构化数据或非结构化数据
elasticsearch安装
官网下载:https://www.elastic.co/downloads/elasticsearch解压
linux运行:./bin/elasticsearch
访问:http://localhost:9200/?pretty
表示安装成功
ES与传统数据的区别
关系型数据库:databases->tables->rows->columns
ES:indices->types->documents->Fileds
index可以理解成数据库 type可以理解为表 doument可以理解为行 filed可以理解为列
索引名词=数据库 索引动词可以理解为数据库的insert语句 索引文档如果存在 旧的将覆盖新的 ES为字段建立的索引为倒排索引
索引CRUD
查看索引 curl localhost:9200/_cat/indices?v 删除索引,通配符形式 curl -XDELETE localhost:9200/索引* 索引起别名 curl -XPUT localhost:9200/索引/_alias/别名 查看别名 curl -XPUT localhost:9200/_cat/aliases?v
建立索引
现在我们建立1个产品的索引(insert)
posman put请求
http://127.0.0.1:9200/opcmdev/product/1
body:
{ "productName": "法瑞思 天然椰棕床垫棕垫硬床垫定做 薄乳胶棕榈榻榻米床垫定制折叠1.2/1.5/1.8", "price": 10, "remark": "不错的床垫", "tags": ["家具", "床垫", "棉花"]
}
我们可以理解为在opcmdev数据库 product表 插入了一条产品信息 方便后面测试 多添加几条数据
http://127.0.0.1:9200/opcmdev/product/2
{ "productName": "南极人（Nanjiren）床垫家纺 加厚立体床垫羽丝绒床垫床褥子学生宿舍", "price": 109, "remark": "不错的床垫", "tags": ["家具", "床垫", "棉花被"] }
http://127.0.0.1:9200/opcmdev/product/3
{ "productName": "￥99.00志高（CHIGO）电火锅家用多功能电炒锅电煮锅不粘多用途锅电热锅 ZG-RH2", "price": 99, "remark": "平底锅", "tags": ["电器", "平底锅"] }
简单的检索文档(查询)
根据id查询
get请求:http://127.0.0.1:9200/opcmdev/product/3
不带条件的简单搜索
get请求:http://127.0.0.1:9200/opcmdev/product/_search 默认返回前10条
{ "took": 1, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 3, "max_score": 1, "hits": [ { "_index": "opcmdev", "_type": "product", "_id": "2", "_score": 1, "_source": { "productName": "南极人（Nanjiren）床垫家纺 加厚立体床垫羽丝绒床垫床褥子学生宿舍", "price": "109", "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花被" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "1", "_score": 1, "_source": { "productName": "法瑞思 天然椰棕床垫棕垫硬床垫定做 薄乳胶棕榈榻榻米床垫定制折叠1.2/1.5/1.8", "price": "10", "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花被" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "3", "_score": 1, "_source": { "productName": "￥99.00志高（CHIGO）电火锅家用多功能电炒锅电煮锅不粘多用途锅电热锅 ZG-RH2", "price": "99", "remark": "平底锅", "tags": [ "电器", "平底锅" ] } } ] } }
通过url?q模糊搜索
http://127.0.0.1:9200/opcmdev/product/_search?q=productName:锅
DSL查询语句
查询条件分词查询
doman spcific language 能够完整更强大的查询功能
默认情况下会将相识度最高的 排在前面 比如搜索平底锅 那么平底锅会排在前面 而其他锅则排在后面
get请求:http://127.0.0.1:9200/opcmdev/product/_search
body:
{ "query":{ "match":{ "productName":"丝绒床垫" } } }
结果
{ "took": 1, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 2, "max_score": 1.5142827, "hits": [ { "_index": "opcmdev", "_type": "product", "_id": "2", "_score": 1.5142827, "_source": { "productName": "南极人（Nanjiren）床垫家纺 加厚立体床垫羽丝绒床垫床褥子学生宿舍", "price": 109, "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花被" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "1", "_score": 0.9389185, "_source": { "productName": "法瑞思 天然椰棕床垫棕垫硬床垫定做 薄乳胶棕榈榻榻米床垫定制折叠1.2/1.5/1.8", "price": 10, "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花" ] } } ] } }
查询条件不分词查询
只需要将条件修改为
{ "query":{ "match_phrase":{ "productName":"丝绒床垫" } } }
结果
{ "took": 7, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 1, "max_score": 1.1507283, "hits": [ { "_index": "opcmdev", "_type": "product", "_id": "2", "_score": 1.1507283, "_source": { "productName": "南极人（Nanjiren）床垫家纺 加厚立体床垫羽丝绒床垫床褥子学生宿舍", "price": 109, "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花被" ] } } ] } }
范围查询
需求:查询价格大于100的产品
get请求:http://127.0.0.1:9200/opcmdev/product/_search
5.*之前版本
{ "query" : { "filtered" : { "filter" : { "range" : { "price" : { "gt" : 30 } } }, "query" : { "match" : { "productName" : "锅" } } } } }
5.*之后版本
{ "query" : { "bool" : { "filter" : { "range" : { "price" : { "gt" : 100 } } }, "must" : { "match" : { "productName" : "锅" } } } } }
结果
为什么结果不对呢
请求：http://127.0.0.1:9200/opcmdev/product/_mapping?pretty
结果：
{ "opcmdev": { "mappings": { "product": { "properties": { "price": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "productName": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "proudctName": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "remark": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "tags": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } } } } }
因为创建索引没有指定mapping(数据类型)默认是text导致无效 但是又不能修改数据类型参考
mapping创建参考:
https://blog.csdn.net/napoay/article/details/52012249
如果正式遇到这种情况 需要修改类型 参考
http://www.cnblogs.com/Creator/p/3722408.html
高亮搜索
get请求:http://127.0.0.1:9200/opcmdev/product/_search
参数
{ "query": { "match_phrase": { "productName": "床垫" } }, "highlight": { "fields": { "productName": {} } } }
结果
{ "took": 19, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 2, "max_score": 0.90414375, "hits": [ { "_index": "opcmdev", "_type": "product", "_id": "2", "_score": 0.90414375, "_source": { "productName": "南极人（Nanjiren）床垫家纺 加厚立体床垫羽丝绒床垫床褥子学生宿舍", "price": 109, "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花被" ] }, "highlight": { "productName": [ "南极人（Nanjiren）<em>床</em><em>垫</em>家纺 加厚立体<em>床</em><em>垫</em>羽丝绒<em>床</em><em>垫</em>床褥子学生宿舍" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "1", "_score": 0.90414375, "_source": { "productName": "法瑞思 天然椰棕床垫棕垫硬床垫定做 薄乳胶棕榈榻榻米床垫定制折叠1.2/1.5/1.8", "price": 10, "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花" ] }, "highlight": { "productName": [ "法瑞思 天然椰棕<em>床</em><em>垫</em>棕垫硬<em>床</em><em>垫</em>定做 薄乳胶棕榈榻榻米<em>床</em><em>垫</em>定制折叠1.2/1.5/1.8" ] } } ] } }
前端显示我们给em这是红色 就行了
分析
aggregations聚合
类似sql的goup by
5.0之前版本
{ "aggs":{ "all_tags":{ "terms":{ "field":"tags" } } } }
5.0之后版本会报错
Fielddata is disabled on text fields by default. Set fielddata=true
因为5.x以后聚合这些操作用单独的数据结构(fielddata)缓存到内存里了 会占用大量堆空间 对性能有影响 默认没有开启
put请求:http://127.0.0.1:9200/opcmdev/_mapping/product/?pretty
结果:
{ "acknowledged": true }
然后我们就可以得到聚合结果
{ "took": 44, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 3, "max_score": 1, "hits": [ { "_index": "opcmdev", "_type": "product", "_id": "2", "_score": 1, "_source": { "productName": "南极人（Nanjiren）床垫家纺 加厚立体床垫羽丝绒床垫床褥子学生宿舍", "price": 109, "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花被" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "1", "_score": 1, "_source": { "productName": "法瑞思 天然椰棕床垫棕垫硬床垫定做 薄乳胶棕榈榻榻米床垫定制折叠1.2/1.5/1.8", "price": 10, "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "3", "_score": 1, "_source": { "productName": "￥99.00志高（CHIGO）电火锅家用多功能电炒锅电煮锅不粘多用途锅电热锅 ZG-RH2", "price": 99, "remark": "平底锅", "tags": [ "电器", "平底锅" ] } } ] }, "aggregations": { "all_tags": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 2, "buckets": [ { "key": "具", "doc_count": 2 }, { "key": "垫", "doc_count": 2 }, { "key": "家", "doc_count": 2 }, { "key": "床", "doc_count": 2 }, { "key": "棉", "doc_count": 2 }, { "key": "花", "doc_count": 2 }, { "key": "器", "doc_count": 1 }, { "key": "平", "doc_count": 1 }, { "key": "底", "doc_count": 1 }, { "key": "电", "doc_count": 1 } ] } } }
可以看到 聚合的字段被分词了 不需要分词需要给参数加上keyword
{ "aggs":{ "all_tags":{ "terms":{ "field":"tags.keyword" } } } }
结果
{ "took": 10, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 3, "max_score": 1, "hits": [ { "_index": "opcmdev", "_type": "product", "_id": "2", "_score": 1, "_source": { "productName": "南极人（Nanjiren）床垫家纺 加厚立体床垫羽丝绒床垫床褥子学生宿舍", "price": 109, "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花被" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "1", "_score": 1, "_source": { "productName": "法瑞思 天然椰棕床垫棕垫硬床垫定做 薄乳胶棕榈榻榻米床垫定制折叠1.2/1.5/1.8", "price": 10, "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "3", "_score": 1, "_source": { "productName": "￥99.00志高（CHIGO）电火锅家用多功能电炒锅电煮锅不粘多用途锅电热锅 ZG-RH2", "price": 99, "remark": "平底锅", "tags": [ "电器", "平底锅" ] } } ] }, "aggregations": { "all_tags": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "家具", "doc_count": 2 }, { "key": "床垫", "doc_count": 2 }, { "key": "平底锅", "doc_count": 1 }, { "key": "棉花", "doc_count": 1 }, { "key": "棉花被", "doc_count": 1 }, { "key": "电器", "doc_count": 1 } ] } } }
如果聚合前需要查询条件 则改参数
{ "query":{ "match":{ "productName":"平底锅" } }, "aggs":{ "all_tags":{ "terms":{ "field":"tags.keyword" } } } }
结果:
{ "took": 1, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 1, "max_score": 0.51040375, "hits": [ { "_index": "opcmdev", "_type": "product", "_id": "3", "_score": 0.51040375, "_source": { "productName": "￥99.00志高（CHIGO）电火锅家用多功能电炒锅电煮锅不粘多用途锅电热锅 ZG-RH2", "price": 99, "remark": "平底锅", "tags": [ "电器", "平底锅" ] } } ] }, "aggregations": { "all_tags": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "平底锅", "doc_count": 1 }, { "key": "电器", "doc_count": 1 } ] } } }
比如再进行二次统计 上面结果每个tag的平均价格 因为price是text类型 就不演示结果了
{ "aggs":{ "all_tags":{ "terms":{ "field":"tags.keyword" }, "aggs":{ "avg_price":{ "avg":{"field":"price"} } } } } }
检索响应各字段意思
{ "took": 1,//这个请求耗费的时间（毫秒） "timed_out": false,//超时是否 一般不会超时 如果响应速度比完整性更重要可以设置超时时间 es会返回超时前收集到的数据 "_shards": {//检索分片的信息 "total": 5,//参与分片的数量 "successful": 5,//有多少成功的 "skipped": 0, "failed": 0//有多少失败的 }, "hits": { "total": 3,//查询总数 "max_score": 1,//指的是所以文档中 相似度最大的值 "hits": [ { "_index": "opcmdev",//索引 "_type": "product",//type "_id": "2",//id "_score": 1,//查询条件的相似度默认最大的排最前面 这个json没有查询条件所有都是1 "_source": {//数据 "productName": "南极人（Nanjiren）床垫家纺 加厚立体床垫羽丝绒床垫床褥子学生宿舍", "price": "109", "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花被" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "1", "_score": 1, "_source": { "productName": "法瑞思 天然椰棕床垫棕垫硬床垫定做 薄乳胶棕榈榻榻米床垫定制折叠1.2/1.5/1.8", "price": "10", "remark": "不错的床垫", "tags": [ "家具", "床垫", "棉花被" ] } }, { "_index": "opcmdev", "_type": "product", "_id": "3", "_score": 1, "_source": { "productName": "￥99.00志高（CHIGO）电火锅家用多功能电炒锅电煮锅不粘多用途锅电热锅 ZG-RH2", "price": "99", "remark": "平底锅", "tags": [ "电器", "平底锅" ] } } ] } }
