ELK之filebeat替代logstash收集日志

filebeat->redis->logstash->elasticsearch

官网下载地址:https://www.elastic.co/downloads/beats/filebeat

  Filebeat是轻量级单用途的日志收集工具,用于在没有安装java的服务器上专门收集日志,可以将日志转发到logstash、elasticsearch或redis等场景中进行下一步处理.

1.Filebeat安装和配置

ip:10.0.0.33

cd /usr/local/src/
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.4.0-x86_64.rpm
grep -vE "#|^$" /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
    - /var/log/messages
  exclude_lines: ['^DBG',"^$"]
  document_type: filesystem-log-0033
output.file:
  path: "/tmp"
  filename: "filebeat.txt"
# exclude_lines:排除以DBG开头和空行
# document_type:设置类型,相当于给每条日志打个标签
systemctl restart filebeat
echo "filebeat has been restarted" >> /var/log/messages
tail -1 /var/log/messages
{"@timestamp":"2019-02-09T12:15:58.454Z","beat":{"hostname":"linux-elk2","name":"linux-elk2","version":"5.4.0"},
"input_type":"log","message":"filebeat has been restarted","offset":130373,
"source":"/var/log/messages","type":"filesystem-log-0033"}

2.配置filebeat输出到redis

cd /usr/local/redis/
vim redis.conf 
bind 10.0.0.33
daemonize yes
save ""
#save 900 1
#save 300 10
#save 60 10000
requirepass 123456
# 启动redis
redis-server /usr/local/redis/redis.conf

vim /etc/filebeat/filebeat.yml 
#修改output
output.redis:
  hosts: "10.0.0.33"
  db: "2"
  port: "6379"
  password: "123456"
  key: "filesystem-log-0033"

systemctl restart filebeat
echo "123456" >> /var/log/messages

redis-cli -h 10.0.0.33 -a 123456

3.配置linux-elk1节点的logstash收取redis中的数据

vim redis-logstash.conf 
input {
    redis {
        data_type => "list"  
        host => "10.0.0.33"
        db => "2"
        port => "6379"
        password => "123456"
        key => "filesystem-log-0033"
    }
}

output {
  if [type] == "filesystem-log-0033" {
    elasticsearch {
        hosts => ["10.0.0.22:9200"]
        index => "filesystem-log-0033-%{+YYYY.MM.dd}"
    }
  }
}
systemctl restart logstash

此时elk2上redis中的数据已经被elk1上的logstash取走,并存到es上了

4.监控Redis的队列长度

# centos7上默认的python版本是2.7,可以用yum下载pip
yum -y install python-pip
pip install redis 

cat  redis-test.py 
#!/usr/bin/env python
import redis
def redis_conn():
    pool=redis.ConnectionPool(host="10.0.0.33",port=6379,db=2,password=123456)
    conn = redis.Redis(connection_pool=pool)
    data = conn.llen('filesystem-log-0033')
    print(data)
redis_conn()

 

filebeat代替logstash收集日志:http://blog.51cto.com/jinlong/2056598