{
"name" : "node-1",
"cluster_name" : "hyxt_logs_es",
"cluster_uuid" : "l_31dzUURiWnLUAI60Zm8w",
"version" : {
"number" : "6.1.2",
"build_hash" : "5b1fea5",
"build_date" : "2018-01-10T02:35:59.208Z",
"build_snapshot" : false,
"lucene_version" : "7.1.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
表示安装配置成功。

常用基本命令：
查询所有数据：curl http://192.168.1.228:9200/_search?pretty

集群健康状态：curl -XGET http://192.168.1.228:9200/_cluster/health?pretty

删除所有数据：curl -X DELETE 'http://192.168.1.228:9200/_all'

删除指定索引：curl -X DELETE 'http://192.168.1.228:9200/索引名称'

注：这里最好不要使用localhost，因为我们需要远程访问，所以，我们应该直接使用对应服务器的ip地址

elasticsearch 集群部署：
elasticsearch 配置集群很简单，基本只要简单修改。我们修改elasticsearch.yml文件：

使得：cluster.name 相同和 node.name 唯一就ok了！

搭建Logstash
安装
解压logstash tar.gz

配置
进入安装目录，为logstash新建一个配置文件config/hy_logstash.conf：

vi hy_logstash.conf

配置文件内容如下

input {

beats {

  port => 5044

    # sincedb_path => "/app/logstash-5.1.1/.sincedb"

}

filter {

if [type] == "app_test" {

grok {

 match => { "message" => "((?<logdate>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})) %{WORD:level} (?<srcCode>\[(?:[a-zA-Z0-9-])+\]\[(?:[a-zA-Z0-9-\.])+:%{NUMBER}\]) - )?(?<srcLog>.+)"  }

}

mutate {remove_field => [ "@timestamp", "@version", "message" ]  }

} else if [type] == "mysql_test" {

grok {

match => { "message" => "((?<logdate>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})) %{WORD:level} (?<srcCode>\[(?:[a-zA-Z0-9-])+\]\[(?:[a-zA-Z0-9-\.])+:%{NUMBER}\]) - )?(?<srcLog>.+)" }

}

mutate {remove_field => [ "@version", "message" ] }

} else if [type] == "nginx_access_test" {

grok {

 match => { "message" => "MAINNGINXLOG %{COMBINEDAPACHELOG} %{QS:x_forwarded_for}" }

# mutate {remove_field => [ "@timestamp", "@version", "message" ]  }

}

date {

match => ["logdate", "yyyy-MM-dd HH:mm:ss.SSS"]

}

if "_grokparsefailure" in [tags] {

} else {

mutate {remove_field => [ "logdate", "@version", "message" ] }

}

if !([level]) {

mutate {

add_field => { "level" => "other" }

}

output {

if "_grokparsefailure" in [tags] {

  file { path => "/var/log/logstash/grokparsefailure-%{type}-%{+YYYY.MM.dd}.log" }

}

elasticsearch {

    hosts => ["ES_IP:9200"]

    index => "hy_log_%{type}-%{+YYYY.MM.dd}"

    document_type => "%{type}"

    template_overwrite => true

}

}

配置文件必须要以”.conf“以尾缀。上面的文件部分配置了应用日志，mysql日志，nginx访问日志，nginx错误日志

ES_IP 为安装elasticsearch 服务器的ip

启动
nohup ./bin/logstash -f config/ hy_logstash.conf &

搭建Kibana
安装
解压kibana tar.gz

配置
进入安装目录，默认配置文件为：config/kibana.yml。可以修改kibana.yml必须要正确配置elasticsearch的地址和索引：

主要修改的地方：

server.host: "192.168.1.228"

elasticsearch.url: "http://192.168.1.228:9200"

启动
nohup ./bin/kibana &。

测试验证
登录访问：http://192.168.1.228:5601(默认端口为5601)

搭建FileBeat
安装
解压filebeat tar.gz

配置
进入安装目录，默认配置文件为：filebeat.yml：

配置如下：

type: log

filebeat.prospectors:

type: log # 应用日志

fields_under_root: true

fields:

app_id: el_shop_1

type: app_test

paths:
- /www/tomcat-demo/logs/catalina.out
multiline: # 多行合并，将\s+at \w开头的行追加到前一行末尾

pattern: '^\s+at \w'

negate: false

match: after
type: log # mysql慢查询日志

fields_under_root: true

fields:

app_id: mysql-slow

type: beat_test

paths:
- /usr/local/filebeat-6.1.2-linux-x86_64/slow.log
multiline: # 多行合并

pattern: '^# User@Host: '

negate: true

match: after

timeout: 10s
type: log # nginx访问日志

fields_under_root: true

fields:

app_id: nginx

type: nginx_access

paths:
- /usr/local/filebeat-6.1.2-linux-x86_64/access.log
multiline: # 多行合并

pattern: '^(\d{1,3}){4} - - ['

negate: true

match: after

timeout: 10s
type: log # nginx错误日志

fields_under_root: true

fields:

app_id: nginx

type: nginx_error

paths:
- /usr/local/filebeat-6.1.2-linux-x86_64/error.log
multiline: # 多行合并

pattern: '^\d{4}/\d{2}/\d{2} '

negate: true

match: after

timeout: 10s

output.logstash:

enabled: true

hosts: ["192.168.1.228:5044"]

index: app-%{[type]}

compression_level: 3

#loadbalance: true

注：

上面的文件部分配置了应用日志，mysql日志，nginx访问日志，nginx错误日志

默认的index 跟踪文件是：data/registry

启动
nohup ./ filebeat &

搭建X-Pack
安装Xpach之前必须先安装好es和kibana

You must install Elasticsearch and Kibana before you install X-Pack. If you plan to use the X-Pack features in Logstash, you must also install Logstash before you install X-Pack

然后分别在es,kibana中安装xpack插件：

es中安装xpack插件
进入es安装目录，执行：
bin/elasticsearch-plugin install x-pack
安装过程中，我们需要进行一些配置，默认统统输入y即可

参考：

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/installing-xpack-es.html

安装完毕需要重启es

kibana中安装xpack插件
进入kibana安装目录，执行：
bin/kibana-plugin install x-pack
参考：

https://www.elastic.co/guide/en/kibana/6.2/installing-xpack-kb.html

安装完毕需要重启kibana

logstash中安装xpack插件（可选）
进入logstash安装目录，执行：
bin/logstash-plugin install x-pack
参考：

https://www.elastic.co/guide/en/logstash/6.2/installing-xpack-log.html

安装完毕需要重启logstash

四、使用Kibana：
Discover 标签页用于交互式探索你的数据