EFK安装调试及使用

elasticsearch注意事项

参考官方文档,安装包下载位置 https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#install-rpm

• 如果单机部署,需要修改配置文件, 注释掉集群配置 /etc/elasticsearch/elasticsearch.yml

# cluster.initial_master_nodes: ["localhost"]

并重启elasticsearch

systemctl restart elasticsearch

使用rpm包安装时, 会自动给elastic用户创建密码, 如果需要手动配置密码,使用如下命令, 期间会让输入密码

[root@localhost ~]# /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i
This tool will reset the password of the [elastic] user.
You will be prompted to enter the password.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: 
Re-enter password for [elastic]: 
Password for the [elastic] user successfully reset.

kibana注意事项

参考官方文档,安装包下载位置 https://www.elastic.co/guide/en/kibana/current/install.html

• 修改中文, 监听地址, kibana和elasticsearch装在同一台主机的话, 连接es的配置不需要修改

i18n.locale: "zh-CN"
server.host: "0.0.0.0"

• kibana首次启动后需要再es中创建个令牌, 命令如下

[root@localhost ~]# /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token --scope kibana
eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTkyLjE2OC4xLjYzOjkyMDAiXSwiZmdyIjoiYTUwOGY3MTBiNTg5MjMyNmNiNTM5NmY5MjJkZTkxYmZiZTEyNzg3MTBkNjkzMWRkNWJhMzFkYTczMTIyMjI1ZiIsImtleSI6IksyMlRJcElCcWlFcjBvXzJTUjlZOnlBSWNRNGtmUWFLUGFYNjZjWGYyYkEifQ==

• 令牌输入后还需要kibana再提供个验证码, 命令如下

[root@localhost ~]# /usr/share/kibana/bin/kibana-verification-code 
Your verification code is:  090 377 

filebeat注意事项

参考官方文档,安装包下载位置 https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html
filebeat通常用于采集应用服务器上的日志, 所以需要部署在对应服务器上

filebeat将日志推送到es时需要使用证书加密,有多种方式,我这块用的指纹, 官方给出的方法如下

到es主机上获取指纹信息

[root@localhost ~]# openssl x509 -fingerprint -sha256 -noout -in /etc/elasticsearch/certs/http_ca.crt | awk --field-separator="=" '{print $2}' | sed 's/://g'

A508F710B5892326CB5396F922DE91BFBE1278710D6931DD5BA31DA73122225F

配置文件示例

filebeat的过滤器不如logstash强大,但是简单的切段日志还是完全够用的,使用tokenizer过滤器

[root@master ~]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: filestream
  paths:
    - /usr/local/openresty/nginx/logs/access.log
  fields:
    log_source: "nginx-access"
  fields_under_root: true
  processors:
  - dissect:
      tokenizer: "%{http_x_forwarded_for} - %{remote_user} %{remote_addr|ip} [%{time_local}] %{request_method} %{uri} %{args} %{status|integer} %{body_bytes_sent|integer} %{http_referer} %{http_user_agent}"
      field: "message"
      target_prefix: "openresty"
      
- type: filestream
  paths:
    - /usr/local/openresty/nginx/logs/error.log
  fields:
    log_source: "nginx-error"
  fields_under_root: true
  processors:
  - dissect:
      tokenizer: "%{date} %{time} [%{level}] %{pid} %{message}"
      field: "message"
      target_prefix: "openresty"
      
- type: filestream
  paths:
    - /data/lcs/logs/lcs.log
  #添加多行日志的处理规则
  parsers:
  - multiline:
      type: pattern
      pattern: '^\['
      negate: true
      match: after
  fields:
    log_source: "lcs"
  fields_under_root: true

  processors:
  - dissect:
      when:
        contains:
          message: "ERROR"
      tokenizer: "[%{time_local}] [%{thread}] [%{traceId}] %{level} %{packge_name_simple} - %{error}"
      field: "message"
      target_prefix: "lcs"
      ignore_failure: true
  - dissect:
      when:
        not.contains:
          message: "ERROR"
      tokenizer: "[%{time_local}] [%{thread}] [%{traceId}] %{level} %{packge_name_simple} - %{app} %{remote_addr|ip} %{user} %{method} %{packge_name} %{class_name} %{interface} %{notes} %{in_params} %{out_params}"
      field: "message"
      target_prefix: "lcs"
      ignore_failure: true

  # - decode_json_fields:
      # fields: ["lcs.in_params", "lcs.out_params"]
      # process_array: true
      # max_depth: 1
      # target: ""
      # overwrite_keys: true
      # add_error_key: true
      # expand_keys: true


output.elasticsearch:
  hosts: ["https://192.168.1.63:9200"]
  preset: balanced
  username: "elastic"
  password: "elastic"
  ssl:
    enabled: true
    ca_trusted_fingerprint: "A508F710B5892326CB5396F922DE91BFBE1278710D6931DD5BA31DA73122225F"
  indices:
    - index: "lcs-logs-%{+yyyy.MM.dd}"
      when.equals:
        log_source: "lcs"
    - index: "nginx-access-logs-%{+yyyy.MM.dd}"
      when.equals:
        log_source: "nginx-access" 
    - index: "nginx-error-logs-%{+yyyy.MM.dd}"
      when.equals:
        log_source: "nginx-error" 


setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false