使用redis缓解elasticsearch压力

使用缓存服务来缓解ES压力

架构如下：
nginx-json-> filebeat --> redis <--logstash ---> elasticsearch <---kibana

1.安装配置redis

yum install redis -y
systemctl start redis 
redis-cli set k1 v1 
redis-cli get k1 

2.配置filebeat

filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true 
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.redis:
  hosts: ["172.16.1.51"]
  keys:
    - key: "nginx_access"
      when.contains:
        tags: "access"
    - key: "nginx_error"
      when.contains:
        tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx_*"
setup.template.enabled: false
setup.template.overwrite: true

3.确保nginx日志为json格式

>/var/log/nginx/access.log 
ab -c 10 -n 100 http://localhost/oldxu

4.启动filebeat并测试是否能存到redis里

systectl restart filebeat

redis-cli
keys * 
TYPE nginx_access 
LLEN nginx_access 
LRANGE nginx_access 1 2 

5.安装配置logstash

[root@db01 ~]# cat /etc/logstash/conf.d/redis.conf 
input {
  redis {
    host => "172.16.1.51"
    port => "6379"
    db => "0"
    key => "nginx_access"
    data_type => "list"
  }
  redis {
    host => "10.0.0.51"
    port => "6379"
    db => "0"
    key => "nginx_error"
    data_type => "list"
  }
}

filter {
  mutate {
    convert => ["upstream_time", "float"]
    convert => ["request_time", "float"]
  }
}

output {
   stdout {}
   if "access" in [tags] {
      elasticsearch {
        hosts => "http://localhost:9200"
        manage_template => false
        index => "nginx_access-%{+yyyy.MM}"
      }
    }
    if "error" in [tags] {
      elasticsearch {
        hosts => "http://localhost:9200"
        manage_template => false
        index => "nginx_error-%{+yyyy.MM}"
      }
    }
}

6.启动Logstash

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis.conf

7.检查redis里是否被取走了

redis-cli
LLEN nginx_access 

8.es-head和kibana查看