oldboy es和logstash
logstash:
input:https://www.elastic.co/guide/en/logstash/current/input-plugins.html
input {
file {
path =>"/var/log/messages"
type => "system"
start_position =>"beginning"
}
file {
path =>"/var/log/elasticsearch/alex.log"
type => "es-error"
start_position =>"beginning"
}
}
output:https://www.elastic.co/guide/en/logstash/current/output-plugins.html
output {
if [type] == "system" {
elasticsearch {
hosts=>["192.168.1.1:9200"]
index=>"system-%{+YYYY.MM.dd}"
}
}
if [type] == "es-error" {
elasticsearch {
hosts=>["192.168.1.1:9200"]
index=>"es-error-%{+YYYY.MM.dd}"
}
}
}
收集java报错堆栈信息，（多行报错）
需要codec plugin
input {
stdin {
codec => multiline {
pattern => "regexp"
negate => "true or false"
what =>"previous or next"//合并到上一行还是下一行
}
}
}
例子1：
input {
stdin {
codec => multiline {
pattern => "^\["
negate => "true"
what =>"previous"
}
}
}
output {
stdout {
codec => "rubydebug"
}
}
案例2：
input {
file {
path =>"/var/log/messages"
type => "system"
start_position =>"begining"
}
file {
path =>"/var/log/elasticsearch/alex.log"
type => "es-error"
start_position =>"beginning"
codec => multiline {
pattern => "^\["
negate => "true"
what =>"previous"//合并到上一行还是下一行
}
}
}
output {
if [type] == "system" {
elasticsearch {
hosts=>["192.168.1.1:9200"]
index=>"system-%{+YYYY.MM.dd}"
}
}
if [type] == "es-error" {
elasticsearch {
hosts=>["192.168.1.1:9200"]
index=>"es-error-%{+YYYY.MM.dd}"
}
}
}
syslog的监听：
logstash-syslog.conf
input {
syslog {
type => "system-syslog"
host => "192.168.56.11"
port => "514" //开启一个进程，并打开514端口
}
}
output {
stdout {
codec => "rubydebug"
}
}
vim /etc/rsyslog.conf
*.* @@192.168.56.11:514 //把所有日志发送到192.168.56.11的514端口
logstash监听tcp端口：
logstash-tcp.conf
input {
tcp {
host => "192.168.56.11"
port => "6666" //监听了6666端口
}
}
output {
stdout {
codec => "rubydebug"
}
}
如果我们用nc，就可以收到
nc 192.168.56.11 6666 < /etc/resolv.conf
或者:echo "alex" > /dev/tcp/192.168.56.11/6666
logstash收集日志和grok
filter块
vim logstash-grok.conf
input {
stdin { }
}
filter {
grok {
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
}
output {
stdout {
codec => "rubydebug"
}
}
logstash收集slowlog日志和grok
logstash-mysql-slow.conf
input {
file {
path =>"/root/slow.log"
type => "msyql-slowlog"
codec => multiline {
pattern =>"^# User@Host:"
negate => true
what => "previous"
}
}
}
filter {
//太多了 没抄完，中间grok
}
output{
stdout =>"rubydebug"
}
logstash 传送到redis，然后另一个logstash从redis取：
input {
stdin{}
}
output {
redis {
host =>"192.168.56.11"
port => "6379"
db =>"6"
data_type=>"list"
key=>"demo"
}
}
在redis中 keys * 可以看到demo
然后 LINDEX demo -1 就能取出来。
llen demo 能知道这个列表里有多少条数据
最后在其他logstash读出来：
input {
redis {
host =>"192.168.56.11"
port => "6379"
db =>"6"
data_type=>"list"
key=>"demo"
}
}
output{
elasticsearch {
hosts=>["192.168.1.1:9200"]
index=>"redis-demo-%{+YYYY.MM.dd}"
}
}
