ELKStack之使用Redis作为消息队列

redis消息队列

安装redis

yum -y install redis

修改配置文件

修改ip

后台运行

启动

systemctl start redis

查看

lsof -i:6379

连接

redis-cli -h 10.13.85.9

 

cd /etc/logstash/conf.d/

vim redis.conf

?

1 2 3 4 5 6 7 8 9 10 11 12 13

input{     stdin {} }   output{     redis{         host => "10.13.85.9"     port => "6379"     db => "6"     data_type => "list"     key => "demo"     } }

启动

/opt/logstash/bin/logstash -f redis.conf

另外开一个窗口启动redis连接

redis-cli -h 10.13.85.9

select 6

验证可以写一个收集apache日志的配置文件

vim apache.conf

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

input{     file{         path => "/var/log/httpd/access_log"         start_position => "beginning"     } } output{     redis{         host => "10.13.85.9"         port => "6379"         db => "6"         data_type => "list"         key => "apache-accesslog"     } }

启动

/opt/logstash/bin/logstash -f apache.conf

查看最好一行

 

生产中可以在另外一台服务器启动一个logstash收集redis里面的数据

在另外服务器上面

vim /etc/logstash/conf.d/indexer.conf

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

input{     redis{         host => "10.13.85.9"     port => "6379"     db => "6"     data_type => "list"     key => "demo"     }   }   output{     stdout{         codec => rubydebug     } }

启动如果成功了加filter处理apache

 

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

input{     redis{         host => "10.13.85.9"     port => "6379"     db => "6"     data_type => "list"     key => "demo"     } }   filter{     grok{         match => { "message" => "%{COMBINEDAPACHELOG}" }     }   }     output{     elasticsearch{         hosts => ["10.13.85.9:9200"]     index => "apache-accesslog-%{+YYYY.MM.dd}"     } }

启动

/opt/logstash/bin/logstash  -f indexer.conf

作业：消息队列kafka