ElasticSearch python入门
官网手册：https://elasticsearch-py.readthedocs.io/en/master/api.html
elasticsearch_dsl手册：https://elasticsearch-dsl.readthedocs.io/en/latest/
安装
pip3 install elasticsearch pip3 install elasticsearch_dsl
连接ES
es = Elasticsearch( ['192.168.38.128:9200'], # 认证信息 # http_auth=('elastic', 'changeme') )
测试连通性
es.ping()
查看版本号
curl -XGET localhost:9200
索引
官网：https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-index_.html
HTTP方式
创建索引（6.0版本，7.x没有类型）
curl -X PUT "localhost:9200/twitter?pretty" -H 'Content-Type: application/json' -d' { "mappings": { "tweet": { "dynamic": "false", "properties": { "counter": { "type": "integer", "store": false }, "tags": { "type": "keyword", "store": true } } } } } '
curl -XGET http://127.0.0.1:9200/_cat/indices?v # 查看所有索引
curl -XDELETE http://127.0.0.1:9200/test_index # 删除索引
客户端方式
创建
方式一：ElasticSearch 7.X版本将移除type
# [types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead es.index(index="my_index", doc_type="test_type", id=0, body={"name": "python", "addr": "深圳"}) es.index(index="my_index", doc_type="test_type", id=1, body={"name": "python", "addr": "深圳"})
方式二：默认创建type=_doc
request_body = { "settings": { "number_of_shards": 3, "number_of_replicas": 2 }, "mappings": { "properties": { "user": {"type": "text"}, "age": {"type": "short"}, "date": {"type": "date"}, "message": {"type": "text"}, } } } es.indices.create(index='test_index', body=request_body)
删除
es.indices.delete('test_index')
文档
创建文档
docs = [ {"user": "user1", "age": 1, "post_date": "2009-11-15T14:12:12", "message": "trying out Elasticsearch"}, {"user": "user2", "age": 2, "post_date": "2010-11-15T14:12:12", "message": "trying out Elasticsearch"}, {"user": "user3", "age": 3, "post_date": "2012-11-15T14:12:12", "message": "trying out Elasticsearch"}, ] ret = [es.create(index='test_index', id=i, body=doc) for i, doc in enumerate(docs)] cpprint(ret)
删除
query = {'query': {'match_all': {}}} # 查找所有文档
query1 = {'query': {'match': {'sex': 'famale'}}} # 删除性别为女性的所有文档
query2 = {'query': {'range': {'age': {'lt': 11}}}} # 删除年龄小于11的所有文档
query3 = {'query': {'term': {'name': 'jack'}}} # 查找名字叫做jack的所有文档
# 删除所有文档
es.delete_by_query(index="test_index", doc_type="_doc", body=query)
查询
es.search(index='test_index', filter_path=['hits.hits._source'])
es.get(index="my_index",doc_type="test_type",id=1)
更新
es.update(index="my_index",doc_type="test_type",id=1,body={"doc":{"name":"python1","addr":"深圳1"}})
查看ES状态
from elasticsearch import Elasticsearch class ES_State(object): def __init__(self, es_hosts): self.es = Elasticsearch(hosts=es_hosts) # 集群健康状态 def get_cluster_health(self, index=None): return self.es.cluster.health(index) # 集群节点及主节点 def get_nodes(self): return self.es.cluster.state(('nodes', 'master_node')) # 集群节点、分片路由及节点路由 def get_routing_table(self, index=None): return self.es.cluster.state(('nodes', 'routing_table'), index) # 索引状态("index","translog","docs","merges","refresh","flush","shards") def get_indices_status(self, index=None): return self.es.indices.stats(index, human=True) # 节点的状态("fs", "http", "indices","jvm", "network", "os", "process", "thread_pool", "transport") def get_nodes_stats(self, node_id=None): return self.es.nodes.stats(node_id, human=True) # 节点的信息 def get_nodes_info(self, node_id=None): return self.es.nodes.info(node_id, human=True) if __name__ == '__main__': es_state = ES_State(es_hosts=[{"host": "127.0.0.1", 'port': "9200"}]) tmp = es_state.get_indices_status() print(tmp)
记一次ElasticSearch 更改 mapping 字段类型的过程
我的个人博客：逐步前行STEP
首先，es不支持直接更改mappinng，所以，更改 mapping 实质上是重建索引。
操作步骤如下：
1、为当前这个索引old_index设置一个别名my_index：
curl -XPOST localhost:9200/_aliases -d ' { "actions": [ { "add": { "alias": "my_index", "index": "old_index" }} ] } '
2、通过别名my_inndex访问索引old_index；
3、重建一个新的索引new_index，在此时使用需要的字段属性；
curl -XPUT localhost:9200/new_index { "mappings": { "doc": { "properties": { "id": { "type": "long" }, "user_id": { "type": "long" } } } }, "settings": { "index": { "number_of_shards": "5", "number_of_replicas": "1" } } }
4、迁移旧的索引old_index数据到新的索引new_index上；
curl -XPOST localhost:9200/_reindex { "source":{ "index":"old_index" }, "dest":{ "index":"new_index" } }
5、为索引new_index设置一个别名为my_index ，同时删除该别名对旧索引old_index的指向：
curl -XPOST localhost:9200/_aliases -d ' { "actions": [ { "remove": { "alias": "my_index", "index": "old_index" }}, { "add": { "alias": "my_index", "index": "new_index" }} ] } '
6、删除索引old_index
curl -XDELETE localhost:9200/old_index
问题汇总
Rejecting mapping update to [xxx] as the final mapping would have more than 1 type: [xxx, xx]
原因：是由于6.0的版本不允许一个index下面有多个type，并且官方说是在接下来的7.0版本中会删掉type
es进行聚合操作时提示Fielddata is disabled on text fields by default
根据es官网的文档执行
GET /megacorp/employee/_search { "aggs": { "all_interests": { "terms": { "field": "interests" } } } }
这个例子时，报错
{ "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [interests] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory." } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [ { "shard": 0, "index": "megacorp", "node": "-Md3f007Q3G6HtdnkXoRiA", "reason": { "type": "illegal_argument_exception", "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [interests] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory." } } ], "caused_by": { "type": "illegal_argument_exception", "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [interests] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory." } }, "status": 400 }
搜了一下应该是5.x后对排序，聚合这些操作用单独的数据结构(fielddata)缓存到内存里了，需要单独开启，官方解释在此fielddata
简单来说就是在聚合前执行如下操作
PUT megacorp/_mapping/employee/ { "properties": { "interests": { "type": "text", "fielddata": true } } }
浙公网安备 33010602011771号