Logstash收集日志写入Redis

     由Logstash收集日志写入Redis，再有Logstash读取Redis在写入elasticsearch

     作为日志缓存介质

     官方文档：https://www.elastic.co/guide/en/logstash/current/plugins-outputs-redis.html

一 配置Logstash写入Redis

1.1.1 配置logstash配置文件

[root@localhost ~]# cat /etc/logstash/conf.d/nginx.conf 
input {
      file {
          path => "/opt/vhosts/fatai/logs/access_json.log"
          start_position => "beginning"
          type => "nginx-accesslog"
          codec => json
          stat_interval => "2"          
      }
      
}


output {
    if [type] == "nginx-accesslog" {
          redis {
             data_type => "list"
         key => "nginx-accesslog-test"
         host => "192.168.10.240"
         port => "6379"
         db => "0"
         password => "123456"
      }

      }
   
}

1.1.2 验证配置文件并重启

[root@localhost ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
Configuration OK
[root@localhost ~]# systemctl restart logstash.service 

1.1.3 检查redis是否有key

 

二 另一台机器配置logstash读取redis文件并写入elasticsearch

[root@DNS-Server tools]# cat /etc/logstash/conf.d/nginx.conf
input {
  redis {
    data_type => "list"
    key => "nginx-accesslog-test"
    host => "192.168.10.240"
    port => "6379"
    db => "0"
    password => "123456"
    codec => "json"
  }

}


output {
    elasticsearch {
      hosts => ["192.168.10.10:9200"]
      index => "logstash-redis-logg-%{+YYYY.MM.dd}"
    }
}

elasticsearch-head验证