ELK logstash grok 匹配系统日志文件
下面将匹配 nginx access.log 和 error.log ，mysql 5.6的慢日志，php 的error.log 以及 php-fpm.log
1、/data/nginx/logs/access_log
access 日志
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
写入测试日志
echo '203.195.212.249 - - [20/Apr/2015:14:37:34 +0800] "GET /v1/coupon/get?access_token=c7e47389ddbe4113a2034798fa5f2084&card_code=http://betawsh.szsllt.com/vikduo/Cards/cancel/784&shop_user_id=103&sign_method=1×tamp=150420144004&sign=96A22ED636A178C1C71291AD7C0C1A0A HTTP/1.0" 200 41 "-" "-" "218.18.0.142"' >> nginx_test_access.log
匹配规则如下：
["message","%{IPORHOST:host_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] \"%{WORD:method} %{URIPATHPARAM:request} %{WORD:procotol}/%{NUMBER:http_version}\" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{QS:http_x_forwarded_for}"]
参考：http://blog.arganzheng.me/posts/use-logstash-to-collect-nginx-access-log.html
另：如果log_format 是完整的
log_format main '$server_name $remote_addr - $remote_user [$time_local] "$request" '
'$status $upstream_status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'$ssl_protocol $ssl_cipher $upstream_addr $request_time $upstream_response_time';
测试日志格式：
www.baidu.com 113.101.146.201 - - [14/Sep/2016:00:00:00 +0800] "GET /api/wxUser?back=http://zy.1pia.com/draws/3?code=8HNQ5F29AH2KVH&code=021XLqf21nzln22u7Vh210lvf21XLqfH&state=20624 HTTP/1.0" 302 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1_2 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Mobile/12B440 MicroMessenger/6.3.22 NetType/WIFI Language/zh_CN" "113.108.146.202" - - 127.0.0.1:9000 0.244 0.244
匹配规则如下：
# /data/logstash/bin/etc/patterns文件中，自定义的 MYTIME
# HOSTPORT %{IPORHOST}:%{POSINT}
filter{
grok{
patterns_dir => "/data/logstash/bin/etc/patterns"
match => [ "message", "(%{IPORHOST:server_name}|-) (%{IPORHOST:remote_addr}|-) - (%{USERNAME:remote_user}|-) \[%{HTTPDATE:time_local}\] \"%{WORD:method} %{URIPATHPARAM:request} %{WORD:procotol}/%{NUMBER:http_version}\" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent} %{QS:http_x_forwarded_for} %{USERNAME:ssl_procotol}? %{USERNAME:ssl_cipher}? (%{HOSTPORT:upstream}|-) (%{NUMBER:resquest_time}|-) (%{NUMBER:response_time}|-)" ]
}
}
2、nginx error.log
/data/nginx/logs/nginx_test_access.log
有两种格式， notice的规则与 warn、error 的不一样
2016/07/13 15:32:38 [notice] 1285#0: signal 17 (SIGCHLD) received
2016/07/13 13:40:17 [warn] 1288#0: *110535 an upstream response is buffered to a temporary file /data/nginx/fastcgi_temp/3/01/0000000013 while reading upstream, client: 10.100.100.35, server: 10.100.200.19, request: "GET /log.php?auth=-curl HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "posapitest.vikduo.com", referrer: "blank"
2016/07/21 13:49:58 [error] 29480#0: *112900 FastCGI sent in stderr: "PHP message: PHP Fatal error: Class 'app\modules\' not found in /data/www/ps/ApiController.php on line 524" while reading response header from upstream, client: 10.20.50.12, server: 10.100.200.19, request: "GET /vaddr?pin_code=100101254&sellemethod=1&sign=8F510437154C1E9DB961CC652D9BCFCF HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.baidu.com"
匹配规则如下：
filter{
grok {
match => { "message" => "(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\]" }
}
if [severity] == 'notice' {
grok {
match => { "message" => "(?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?<client>%{IP}|%{HOSTNAME}))?(?:, server: %{IPORHOST:server})?(?:, request: %{QS:request})?(?:, upstream: \"%{URI:upstream}\")?(?:, host: %{QS:host})?(?:, referrer: \"%{URI:referrer}\")?" }
}
}else{
grok {
match => { "message" => "(?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?<client>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})?(?:, request: %{QS:request})?(?:, upstream: \"%{URI:upstream}\")?(?:, host: %{QS:host})?(?:, referrer: \"%{URI:referrer}\")?" }
}
}
}
3、mysql slow_query.log
# Time: 160627 17:49:41
# User@Host: newwsh_r[newwsh_r] @ [10.207.163.147] Id: 29836306
# Query_time: 5.040486 Lock_time: 0.000113 Rows_sent: 25 Rows_examined: 3645275
SET timestamp=1467014886;
SELECT * FROM `wx_user_infos`
LIMIT 3645250, 25;
或者：
# Time: 160630 13:51:56
# User@Host: baseapi[baseapi] @ [10.104.218.111] Id: 30555642
# Query_time: 0.731235 Lock_time: 0.000064 Rows_sent: 0 Rows_examined: 1
SET timestamp=1467265916;
UPDATE `wx_infos` SET `modified`=1467265915 WHERE `id`=14687666;
匹配规则如下：
(?m)^#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+\[(?:%{IP:clientip})?\]\s+Id:\s+%{NUMBER:id,int}\n#\s+Query_time:\s+%{NUMBER:query_time,float}\s+Lock_time:\s+%{NUMBER:lock_time,float}\s+Rows_sent:\s+%{NUMBER:rows_sent,int}\s+Rows_examined:\s+%{NUMBER:rows_examined,int}\s+(Rows_affected:\s+%{NUMBER:rows_affected,int}\n#\s+Bytes_sent:\s+%{NUMBER:bytes_sent,int}\n\s*)?(?:use\s+%{USER:usedatabase};\s*\n)?SET\s+timestamp=%{NUMBER:timestamp};\n\s*(?<query>(?<action>\w+)\b.*)\s*(?:\n#\s+Time)?.*$
4、php php-fpm.log
[09-May-2016 16:27:04] ERROR: failed to ptrace(PEEKDATA) pid 1804: Input/output error (5)
[22-Apr-2016 23:29:19] NOTICE: ready to handle connections
匹配规则如下：
# /data/logstash/bin/etc/patterns文件中，自定义的 MYTIME
# MYTIME %{MONTHDAY}[./-]%{MONTH}[./-]%{YEAR} %{TIME}
filter{
grok{
patterns_dir => "/data/logstash/bin/etc/patterns"
match => [ "message", "\[%{MYTIME:datetime}\]\s+%{LOGLEVEL:severity}:\s+%{GREEDYDATA:errormessage}" ]
}
}
5、php php-error.log
[02-Feb-2015 10:38:29 Europe/Berlin] PHP Notice: Undefined index: global-showcode in /opt/lampp/htdocs/xampp/showcode.php on line 5
匹配规则如下：
# /data/logstash/bin/etc/patterns文件中，自定义的 MYTIME
# MYTIME %{MONTHDAY}[./-]%{MONTH}[./-]%{YEAR} %{TIME}
filter{
grok{
patterns_dir => "/data/logstash/bin/etc/patterns"
match => [ "message", "\[%{MYTIME:datatime}\s+%{DATA:zone}\]\s+PHP\s+%{LOGLEVEL:severity}:\s+%{GREEDYDATA:errormessage}" ]
}
}
6、php php_slow.log
[20-Sep-2016 10:20:17] [pool www] pid 21690
script_filename = /data/www/web/index.php
[0x00007f9551130828] execute() /data/www/vendor/***/Command.php:838
匹配规则如下：
phpslow \[%{MYTIME:time_local}\] \[pool %{SSL:pool}\] pid %{SSL:pid}\n%{SS:content}
SSL %{USERNAME}
SS ([a-zA-Z0-9._-]|\s|\[|\]|\=|\/|\(|\)|\:)+
SSL %{USERNAME}
SS ([a-zA-Z0-9._-]|\s|\[|\]|\=|\/|\(|\)|\:)+
filter {
multiline { pattern => "^\[0$" negate => true what => "previous" }
grok {
patterns_dir => "./patterns"
match => {"message" => "%{phpslow}"}
}
}
可用此地址测试: http://grokdebug.herokuapp.com/
