elk笔记2.0

curl 'localhost:9200/_cat/indices?v'

查看所有索引

 

curl -XDELETE http://localhost:9200/*

清除所有索引

 

1、filebeat配置：

filebeat读取日志文件写入redis，为了保持日志采集、解析和持久化过程中的一致性，对同一种类型日志使用相同解析规则，在日志采集时添加自定义字段log_typ（日志类型）和log_file（日志来源）。

filebeat.inputs:
- type: log
  paths:
    - /home/public/pm2/channelHandle-out-2.log
  fields:
    log_file: xsj-channelhandle-out
    log_type: a-out-log
  fields_under_root: true
  encoding: utf-8
- type: log
  paths:
    - /home/public/pm2/channelHandle-err-2.log
  fields:
    log_file: xsj-channelhandle-err
    log_type: a-err-log
  fields_under_root: true
  encoding: utf-8

processors:
  - drop_event:
      when.not.contains:
        message: "收到"
output.redis:
  hosts: ["10.0.1.223:6700","10.0.1.224:6700"]
  db: 0
  password: "1234@abcd.com"
  key: "%{[log_type]:api}"
  timeout: 5

./filebeat -e

2、logstatsh

logstash规则解析：

创建多个配置文件，一个配置文件用来完成一种类型日志的消费、解析和持久化工作。

cd /data/logstash/
mkdir logstash.d
touch logstash.d/{channelhandle.conf,wss-nginx.conf}
vim logstash.d/channelhandle.conf

# 配置内容如下

input {
  redis {
    host => "127.0.0.1"
    port => "6700"
    password => "1234@abcd.com"
    data_type => "list"
    key => "a-out-log"
  }
  redis {
    host => "127.0.0.1"
    port => "6700"
    password => "1234@abcd.com"
    data_type => "list"
    key => 'a-err-log'
  }
 }

filter {
  mutate {
     rename => {"[host][name]" => "host_name" }
     remove_field => ["ecs", "input", "log", "agent", "host"]
     }
  if [log_type] == "a-out-log" {
     grok {
        match => {
          "message" => ["(?<recTime>(\d+-){2}\d+\s+(\d+:){2}\d+).*?收到.*?(?<content>{.*})", "^收到.*?(?<content>{.*})"]
        }
     }
     json {
        source => "content"
        remove_field => ["content"]
     }
     mutate {
       remove_field => ["content", "param"]
     }
     date {
       match => [ "reqTime", "yyyy-MM-dd HH:mm:ss", "UNIX"]
       timezone => "Asia/Shanghai"
       target => ["@timestamp"]
     }
     if [recTime] {
        date {
           match => ["recTime", "yyyy-MM-dd HH:mm:ss", "UNIX"]
           target => ["recTime"]
        }
        ruby {
           init => "require 'time'"
           code => "duration = (event.get('recTime') - event.get('@timestamp')); event.set('duration', duration)"
        }
        mutate { remove_field => ["recTime"] }
     }
     geoip {
        source => "ip"
        target => "geoip"
        add_tag => [ "agent-ip" ]
        }
  }
}
output {
  elasticsearch {
    hosts => ["10.0.1.221:9200", "10.0.1.222:9200"]
    index => "logstash-%{[log_file]}-%{+YYYYMMdd}"
  }
 }

supervisor进程管理配置文件：

[program:logstash]
command=/data/logstash/bin/logstash
autostart=true
autorestart=true
logfile_maxbytes=50MB
logfile_backups=5
environment=JAVA_HOME=/usr/local/jdk
stdout_logfile=/var/log/supervisor/logstash.out.log
stderr_logfile=/var/log/supervisor/logstash.err.log

/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/nginx.conf  #测试配置文件
Configuration OK

3、 redis

supervisor 进程管理启动文件：

[program:redis]
command=/usr/local/bin/redis-server /data/redis/conf/redis-6700.conf
autostart=true
autorestart=true
logfile_maxbytes=50MB
logfile_backups=5
stdout_logfile=/var/log/supervisor/redis.out.log
stderr_logfile=/var/log/supervisor/redis.err.log

redis 主从配置（redis/config/redis-6700.conf）

slaveof 10.0.1.223 6700
masterauth 1234@abcd.com
slave-serve-stale-data yes
slave-read-only yes

4、java环境配置（elastic\logstash）

[root@server01 src]$ tar -zvxf jdk-8u151-linux-x64.tar.gz -C /data/app/
[root@server01 src]$ ln -s /data/app/jdk1.8.0_151 /data/app/jdk
[root@server01 src]$ cat <<EOF >> /etc/profile
export JAVA_HOME=/data/app/jdk
PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar
EOF
[root@server01 src]$ source /etc/profile

5、elasticsearch

/etc/sysctl.conf

vm.max_map_count = 655350

/etc/security/limits.conf

*  -  nofile 102400
*  -  nproc 4096

修改/etc/supervisord.conf

minfds=102400 

配置子进程管理文件：/etc/supervisord.d/elastic.ini

[program:elasticsearch]
user=elkuser
command=/data/elasticsearch/bin/elasticsearch
environment=ES_HEAP_SIZE=2g
minfds=102400
minprocs=32768
autostart=true
autorestart=true
logfile_maxbytes=50MB
logfile_backups=5
stdout_logfile=/var/log/supervisor/elasticsearch.out.log
stderr_logfile=/var/log/supervisor/elasticsearch.err.log

kibana 汉化： 配置文件修改：i18n.locale: "zh-CN"

6、 logstash 解析wss反向代理nginx日志：

nginx配置文件中定义的access日志格式：

log_format main '$remote_addr $http_X_Forwarded_For [$time_local] '  
                 '$upstream_addr "$upstream_response_time" "$request_time" '  
                 '$http_host $request '  
                 '"$status" $body_bytes_sent "$http_referer" '  
                 '"$http_accept_language" "$http_user_agent" '; 

nginx json日志格式

log_format json escape=json  '{"@timestamp":"$time_iso8601",'
                    '"@source":"$server_addr",'
                    '"hostname":"$hostname",'
                    '"ip":"$http_x_forwarded_for",'
                    '"client":"$remote_addr",'
                    '"request_method":"$request_method",'
                    '"scheme":"$scheme",'
                    '"domain":"$server_name",'
                    '"client_host":"$host",'
                    '"referer":"$http_referer",'
                    '"request":"$request_uri",'
                    '"args":"$args",'
                    '"size":$body_bytes_sent,'
                    '"status": $status,'
                    '"responsetime":$request_time,'
                    '"upstreamtime":"$upstream_response_time",'
                    '"upstreamaddr":"$upstream_addr",'
                    '"http_user_agent":"$http_user_agent",'
                    '"https":"$https"'
                    '}';

自定义解析编码存放位置： /data/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns

WSS_NGINX_GROK
vim wss-nginx

WSS_NGINX_ACCESS %{IPORHOST:clientip} %{NOTSPACE:http_x_forwarded_for} \[%{HTTPDATE:timestamp}\] (?<upstream_addr>%{IPORHOST}:%{NUMBER}) \"%{NUMBER:upstream_response}\" \"%{NUMBER:request_time}\" (?<http_host>%{IPORHOST}:%{NUMBER}) %{NOTSPACE:request} /.*? \"%{NUMBER:status}\" (?:%{NUMBER:sent_bytes}|-) (\".*\"?){2} %{QS:agent}

LEYOU_API_NGINX_GROK
%{IPORHOST:clientip}.*?\[%{HTTPDATE:timestamp}\] \"(?<request>%{WORD}) (?<request_url>.*?) .*?\" %{NUMBER:status} (?:%{NUMBER:sent_bytes}|-) (\".*?)\" %{QS:agent} %{NOTSPACE:http_x_forwarded_for}

filebeat配置：

filebeat.inputs:
- type: log
  paths:
    - /usr/local/nginx/logs/access.log
  fields:
    log_file: lc-wss-access
    log_type: wss-nginx-access
  fields_under_root: true
  encoding: utf-8
output.redis:
  hosts: ["10.0.1.223:6700", "10.0.1.224:6700"]
  db: 0
  password: "1234@abcd.com"
  key: "%{[log_type]:xsj_wss}"
  timeout: 5

logstash配置文件：

input {
  redis {
    host => "127.0.0.1"
    port => "6700"
    password => "1234@abcd.com"
    data_type => "list"
    key => 'wss-nginx-access'
  }
}

filter {
  mutate { 
     rename => {"[host][name]" => "host_name" }
     remove_field => ["ecs", "input", "log", "agent", "host"]
     } 

  if [log_type] == "wss-nginx-access" {
        grok {
            match => [ "message" , "%{WSS_NGINX_ACCESS}"]
            overwrite => [ "message" ]
            remove_tag => ["_grokparsefailure"]
        } 
        mutate {
           convert => ["response", "integer"]
           convert => ["bytes", "integer"]
           convert => ["responsetime", "float"]
         }
        geoip {
           source => "http_x_forwarded_for"
           target => "geoip"
           add_tag => [ "nginx-geoip" ]
        }
        date {
           match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]
           target => ["@timestamp"]
        }
    }

}

output {
  elasticsearch {
    hosts => ["10.0.1.221:9200", "10.0.1.222:9200"]
    index => "logstash-%{[log_file]}-%{+YYYYMMdd}"
  }
 }

配置pipelines启用多配置文件

 - pipeline.id: channelhandle
   pipeline.workers: 1 
   path.config: "/data/logstash/logstash.d/channelhandle.conf"
 - pipeline.id: wss-nginx
   pipeline.workers: 1
   path.config: "/data/logstash/logstash.d/wss-nginx.conf"

地图热力图

/data/logstash-7.0.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.0.1-java/lib/logstash/outputs/elasticsearch/elasticsearch-template-es7x.json