Elasticsearch初探（一）

一、安装（Linux篇）

现在的开源软件越来越成熟，都打好包了，下载后直接解压就可以测试了。

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.1.tar.gz
tar -zxvf elasticsearch-6.3.1.tar.gz 
cd elasticsearch-6.3.1/bin/
./elasticsearch

注意：elasticsearch不允许使用root权限启动，所以使用root下载解压的同学，要chown改变一下文件夹的用户权限。

看到如下信息就说明启动成功了。

[2018-07-12T11:07:00,877][INFO ][o.e.n.Node               ] [] initializing ...
[2018-07-12T11:07:00,922][INFO ][o.e.e.NodeEnvironment    ] [4LdtI3q] using [1] data paths, mounts [[/home (/dev/mapper/cl-home)]], net usable_space [229.9gb], net total_space [240.9gb], types [xfs]
[2018-07-12T11:07:00,923][INFO ][o.e.e.NodeEnvironment    ] [4LdtI3q] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-07-12T11:07:00,938][INFO ][o.e.n.Node               ] [4LdtI3q] node name derived from node ID [4LdtI3q-T2ehl0Z6Xv6cOA]; set [node.name] to override
[2018-07-12T11:07:00,939][INFO ][o.e.n.Node               ] [4LdtI3q] version[6.3.1], pid[21521], build[default/tar/eb782d0/2018-06-29T21:59:26.107521Z], OS[Linux/3.10.0-514.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_112/25.112-b15]
[2018-07-12T11:07:00,939][INFO ][o.e.n.Node               ] [4LdtI3q] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.af3cuwkJ, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home/lings/elastic/elasticsearch-6.3.1, -Des.path.conf=/home/lings/elastic/elasticsearch-6.3.1/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-07-12T11:07:02,794][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [aggs-matrix-stats]
[2018-07-12T11:07:02,794][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [analysis-common]
[2018-07-12T11:07:02,795][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [ingest-common]
[2018-07-12T11:07:02,795][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [lang-expression]
[2018-07-12T11:07:02,795][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [lang-mustache]
[2018-07-12T11:07:02,795][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [lang-painless]
[2018-07-12T11:07:02,795][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [mapper-extras]
[2018-07-12T11:07:02,795][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [parent-join]
[2018-07-12T11:07:02,795][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [percolator]
[2018-07-12T11:07:02,795][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [rank-eval]
[2018-07-12T11:07:02,795][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [reindex]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [repository-url]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [transport-netty4]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [tribe]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-core]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-deprecation]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-graph]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-logstash]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-ml]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-monitoring]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-rollup]
[2018-07-12T11:07:02,796][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-security]
[2018-07-12T11:07:02,797][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-sql]
[2018-07-12T11:07:02,797][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-upgrade]
[2018-07-12T11:07:02,797][INFO ][o.e.p.PluginsService     ] [4LdtI3q] loaded module [x-pack-watcher]
[2018-07-12T11:07:02,797][INFO ][o.e.p.PluginsService     ] [4LdtI3q] no plugins loaded
[2018-07-12T11:07:05,500][INFO ][o.e.x.s.a.s.FileRolesStore] [4LdtI3q] parsed [0] roles from file [/home/lings/elastic/elasticsearch-6.3.1/config/roles.yml]
[2018-07-12T11:07:05,896][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/21588] [Main.cc@109] controller (64 bit): Version 6.3.1 (Build 4d0b8f0a0ef401) Copyright (c) 2018 Elasticsearch BV
[2018-07-12T11:07:06,171][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-07-12T11:07:06,425][INFO ][o.e.d.DiscoveryModule    ] [4LdtI3q] using discovery type [zen]
[2018-07-12T11:07:07,061][INFO ][o.e.n.Node               ] [4LdtI3q] initialized
[2018-07-12T11:07:07,061][INFO ][o.e.n.Node               ] [4LdtI3q] starting ...
[2018-07-12T11:07:07,194][INFO ][o.e.t.TransportService   ] [4LdtI3q] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-07-12T11:07:07,216][WARN ][o.e.b.BootstrapChecks    ] [4LdtI3q] max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2018-07-12T11:07:07,216][WARN ][o.e.b.BootstrapChecks    ] [4LdtI3q] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-07-12T11:07:10,259][INFO ][o.e.c.s.MasterService    ] [4LdtI3q] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {4LdtI3q}{4LdtI3q-T2ehl0Z6Xv6cOA}{y7E-EJNzQKq7Xj5-B-okOA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=16658751488, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-07-12T11:07:10,264][INFO ][o.e.c.s.ClusterApplierService] [4LdtI3q] new_master {4LdtI3q}{4LdtI3q-T2ehl0Z6Xv6cOA}{y7E-EJNzQKq7Xj5-B-okOA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=16658751488, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {4LdtI3q}{4LdtI3q-T2ehl0Z6Xv6cOA}{y7E-EJNzQKq7Xj5-B-okOA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=16658751488, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[2018-07-12T11:07:10,279][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [4LdtI3q] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-07-12T11:07:10,280][INFO ][o.e.n.Node               ] [4LdtI3q] started
[2018-07-12T11:07:10,622][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [4LdtI3q] Failed to clear cache for realms [[]]
[2018-07-12T11:07:10,636][INFO ][o.e.l.LicenseService     ] [4LdtI3q] license [d717e767-fe3f-4b36-bc85-cad6dbde3efc] mode [basic] - valid
[2018-07-12T11:07:10,644][INFO ][o.e.g.GatewayService     ] [4LdtI3q] recovered [2] indices into cluster_state
[2018-07-12T11:07:10,956][INFO ][o.e.c.r.a.AllocationService] [4LdtI3q] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[customer][2], [customer][3]] ...]).

 

 

二、REST使用示例

1. 查看集群的状态
curl -X GET "localhost:9200/_cat/health?v"
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1531365002 11:10:02  elasticsearch yellow          1         1     10  10    0    0       10             0                  -                 50.0%
注意这个status的状态是yellow，理想的状态应该是green，原因是没有配置集群。

2. 查看节点的状态
curl -X GET "localhost:9200/_cat/nodes?v"
ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1           20          57   3    0.07    0.06     0.06 mdi       *      4LdtI3q

3. 查看所有索引（类比mysql的库）的状态
curl -X GET 'http://localhost:9200/_cat/indices?v'
health status index    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   bank     BO1IYzoXSpGGyYP3XKo4bA   5   1       1000            0    475.1kb        475.1kb
yellow open   customer -8lF3GAFT1WI43gR9DiYpw   5   1          5            0     17.7kb         17.7kb
有两个索引bank和customer，一个1000条数据，一个有5条数据。

4. 查看索引（customer）的状态
curl -X PUT "localhost:9200/customer?pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "resource_already_exists_exception",
        "reason" : "index [customer/-8lF3GAFT1WI43gR9DiYpw] already exists",
        "index_uuid" : "-8lF3GAFT1WI43gR9DiYpw",
        "index" : "customer"
      }
    ],
    "type" : "resource_already_exists_exception",
    "reason" : "index [customer/-8lF3GAFT1WI43gR9DiYpw] already exists",
    "index_uuid" : "-8lF3GAFT1WI43gR9DiYpw",
    "index" : "customer"
  },
  "status" : 400
}

5. 更新（不存在时则增加）索引的document（类比Mysql的行）
curl -X PUT "localhost:9200/customer/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
  "name": "John Doe"
}
'
响应如下：
{
  "_index" : "customer",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 8, //每更新一次版本记录就+1
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 7,
  "_primary_term" : 4
}

6. 增加索引的document（和PUT增加的区别就是不用指定document的id，会随机生成一个）
curl -X POST "localhost:9200/customer/_doc?pretty" -H 'Content-Type: application/json' -d'
{
  "name": "Jane Doe"
}
'
响应如下：
{
  "_index" : "customer",
  "_type" : "_doc",
  "_id" : "Bu1tjWQBZa8_pSJjP5AF", //随机生成的id
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 8,
  "_primary_term" : 4
}

7. 查看指定的document
curl -X GET "localhost:9200/customer/_doc/1?pretty"
响应结果
{
  "_index" : "customer",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 8,
  "found" : true,
  "_source" : {
    "name" : "John Doe"
  }
}

8. 删除指定的document
curl -X DELETE "localhost:9200/customer/_doc/B-1ujWQBZa8_pSJjH5CC?pretty"
响应结果：
{
  "_index" : "customer",
  "_type" : "_doc",
  "_id" : "B-1ujWQBZa8_pSJjH5CC",
  "_version" : 3,
  "result" : "not_found",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 7,
  "_primary_term" : 4
}

9. 批量插入文本数据
curl -H "Content-Type: application/json" -XPOST "localhost:9200/bank/_doc/_bulk?pretty&refresh" --data-binary "@/home/lings/accounts.json"
/home/lings/accounts.json为json文件的存放路径


10. 按照account_number升幂排序
curl -X GET "localhost:9200/bank/_search?q=*&sort=account_number:asc&pretty"
完全等效于这个命令
curl -X GET "localhost:9200/bank/_search" -H 'Content-Type: application/json' -d'
{
  "query": { "match_all": {} },
  "sort": [
    { "account_number": "asc" }
  ]
}
'

响应结果：
{
  "took" : 3, //查询花费了3ms
  "timed_out" : false, 
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000, //一共有1000条数据
    "max_score" : null,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "0",
        "_score" : null,
        "_source" : {
          "account_number" : 0,
          "balance" : 16623,
          "firstname" : "Bradshaw",
          "lastname" : "Mckenzie",
          "age" : 29,
          "gender" : "F",
          "address" : "244 Columbus Place",
          "employer" : "Euron",
          "email" : "bradshawmckenzie@euron.com",
          "city" : "Hobucken",
          "state" : "CO"
        },
        "sort" : [
          0
        ]
      },
 ......省略中间数据
       {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "9",
        "_score" : null,
        "_source" : {
          "account_number" : 9,
          "balance" : 24776,
          "firstname" : "Opal",
          "lastname" : "Meadows",
          "age" : 39,
          "gender" : "M",
          "address" : "963 Neptune Avenue",
          "employer" : "Cedward",
          "email" : "opalmeadows@cedward.com",
          "city" : "Olney",
          "state" : "OH"
        },
        "sort" : [
          9  //一共只返回了10个document
        ]
      }
    ]
  }
}

11. 查询住址包含Street单词的所有document
curl -X GET "localhost:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": { "match": {"address":"Street"} },
  "sort": [
    { "account_number": "desc" }
  ]
}
'
响应如下：
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 385, //满足条件的385个，总共还是返回10个document
    "max_score" : null,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "996",
        "_score" : null,
        "_source" : {
          "account_number" : 996,
          "balance" : 17541,
          "firstname" : "Andrews",
          "lastname" : "Herrera",
          "age" : 30,
......


12. 查询住址包含Street单词的降幂排序后的2个document
curl -X GET "localhost:9200/bank/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": { "match": {"address":"Street"} },
  "sort": [
    { "account_number": "desc" }
  ],
  "size":2
}
'
响应如下：
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 385,
    "max_score" : null,
    "hits" : [
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "996",
        "_score" : null,
        "_source" : {
          "account_number" : 996,
          "balance" : 17541,
          "firstname" : "Andrews",
          "lastname" : "Herrera",
          "age" : 30,
          "gender" : "F",
          "address" : "570 Vandam Street",
          "employer" : "Klugger",
          "email" : "andrewsherrera@klugger.com",
          "city" : "Whitehaven",
          "state" : "MN"
        },
        "sort" : [
          996
        ]
      },
      {
        "_index" : "bank",
        "_type" : "_doc",
        "_id" : "994",
        "_score" : null,
        "_source" : {
          "account_number" : 994,
          "balance" : 33298,
          "firstname" : "Madge",
          "lastname" : "Holcomb",
          "age" : 31,
          "gender" : "M",
          "address" : "612 Hawthorne Street",
          "employer" : "Escenta",
          "email" : "madgeholcomb@escenta.com",
          "city" : "Alafaya",
          "state" : "OR"
        },
        "sort" : [
          994
        ]
      }
    ]
  }
}