ELK 之三：Kibana 使用与Tomcat、Nginx 日志格式处理
一：kibana安装：
kibana主要是搜索elasticsearch的数据，并进行数据可视化的展现，新版使用nodejs。
1、下载地址：
https://www.elastic.co/downloads/kibana
2、解压安装：
[root@node6 local]# tar xvf kibana-4.1.1-linux-x64.tar.gz [root@node6 local]# mv kibana-4.1.1-linux-x64 kibana [root@node6 ~]# cd /usr/local/kibana/ [root@node6 kibana]# ls bin config LICENSE.txt node plugins README.txt src
3、编辑配置文件：
[root@node6 kibana]# cd config/ [root@node6 config]# ls kibana.yml [root@node6 config]# vim kibana.yml elasticsearch_url: "http://192.168.10.206:9200"
4、直接启动：
[root@node6 kibana]# bin/kibana {"name":"Kibana","hostname":"node6.a.com","pid":3942,"level":30,"msg":"No existing kibana index found","time":"2016-04-12T12:20:50.069Z","v":0} {"name":"Kibana","hostname":"node6.a.com","pid":3942,"level":30,"msg":"Listening on 0.0.0.0:5601","time":"2016-04-12T12:20:50.096Z","v":0}
5、验证启动:
[root@node6 ~]# ps -ef | grep kibana root 3942 3745 3 20:20 pts/2 00:00:01 bin/../node/bin/node bin/../src/bin/kibana.js root 3968 3947 0 20:21 pts/3 00:00:00 grep kibana [root@node6 ~]# ss -tnl | grep 5601 LISTEN 0 128 *:5601 *:*
6、后台启动：
[root@node6 kibana]# nohup bin/kibana & [1] 3975
7、访问测试：默认监听端口5601
http://192.168.10.206:560１
8、配置索引：索引的名称要和logstash的output生成的索引能进行匹配才可以
9、查看数据：默认显示最新的500个文档
10、数据精确搜索：
11、搜索高级语法：
status:404 OR status:500 #搜索状态是404或者是500之一的 status:301 AND status:200 #搜索即是301和200同时匹配的 status：[200 TO 300] ：搜索指定范围的
12、保存常用的搜索语法：
二：其他的常用模块：
1、系统日志收集---> syslog：配置syslog结果写入到elasticsearch，指定端口514，主机就是要收集日志的服务器IP地址，即可使用
2、访问日志：nginx转换成json格式
3、错误日志：使用codec插件：
https://www.elastic.co/guide/en/logstash/1.5/codec-plugins.html
input { stdin { codec => multiline { #多行日志，比如java的日志 pattern => "^\s" #pattern => ".*\t.*" #找到换行符，会把多行认为是一行，即会把当前行和上一行合成一行，直到有换行符结束 what => "previous" } } }
4、运行日志 codec => json，如果不是json要使用grok进行匹配，相对比较麻烦，如果丢日志就看logstash.log，另外检查日志是否有效的json格式：
json效验地址：http://www.bejson.com/
5、kibana的时区和时间问题:kibana会自动根据浏览器将时间加8小时，通过logstash写入会自动解决，如果通过python脚本等写入会产生时间问题
6、在地图显示IP具体来源地址：
https://www.elastic.co/guide/en/logstash/1.5/filter-plugins.html
7、条件判断：
input { file { type => "apache" path => "/var/log/apache.log" } file { type => "tomcat" path => "/var/log/tomcat.log" } }
filter { if [type] == "apache" { #假如索引为apache，就执行以下操作 redis { data_type => "list" key => "system-message-jack" host => "192.168.10.205" port => "6379" db => "0" } if [type] == "tomcat" { #假如索引为tomcat，就执行一次操作 redis { data_type => "list" key => "system-message-tomcat" host => "192.168.10.205" port => "6379" db => "1" #写不同的数据库 } }
nginx 最好设置buffer大小，64k
kibana要添加elastsearch的key
搜索的语法：直接搜索键值 a:b AND ALL NOT进行匹配。范围 [200-299]
6.测试logstash配置文件语法是否正确：
6.1:配置正确的检查结果：
[root@elk-server2 conf.d]# /etc/init.d/logstash configtest
Configuration OK
6.2：语法错误的显示结果：
[root@elk-server2 tianqi]# /etc/init.d/logstash configtest The given configuration is invalid. Reason: Expected one of #, {, } at line 17, column 53 (byte 355) after output { if [type] == "nginx3" { elasticsearch { hosts => ["192.168.0.251:9200"] index => "logstash-newsmart-nginx3-" {:level=>:fatal} #会指明语法错误的具体地方
三：tomcat日志：
1、tomcat日志默认不是json格式的，但是logstash分析的时候就没有key和valus了，所以我们可以将tomcat日志的格式定义为json的格式：
directory="logs" prefix="localhost_access_log." suffix=".log" pattern="{"client":"%h", "client user":"%l", "authenticated":"%u", "access time":"%t", "method":"%r", "status":"%s", "send bytes":"%b", "Query?string":"%q", "partner":"%{Referer}i", "Agent version":"%{User-Agent}i"}"/>
2、取到的日志结果为：
{"client":"180.95.129.206", "client user":"-", "authenticated":"-", "access time":"[20/Apr/2016:03:47:40 +0000]", "method":"GET /image/android_logo.png HTTP/1.1", "status":"200", "send bytes":"1915", "Query string":"", "partner":"http://mobile.weathercn.com/index.do?id=101160101&partner=1000001003", "Agent version":"Mozilla/5.0 (Linux; U; Android 5.1.1; zh-cn; NX510J Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko)Version/4.0 Chrome/37.0.0.0 MQQBrowser/6.6 Mobile Safari/537.36"}
3、在线验证是否合法的json格式：
地址：http://www.bejson.com/，将完整的一行日志复制到验证框，然后点验证即可：结果如下
四：nginx 日志格式处理：
1、编辑nginx.conf配置文件，自定义一个日志格式：
[root@node5 ~]# vim /etc/nginx/nginx.conf
2、添加内容如下：
log_format logstash_json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"url":"$uri",' '"domain":"$host",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"agent":"$http_user_agent",' '"status":"$status"}';
3、编辑主机配置：
[root@node5 ~]# grep -v "#" /etc/nginx/conf.d/locathost.conf | grep -v "^$" server { listen 9009; #监听的端口 server_name www.a.com; #主机名 access_log /var/log/nginx/json.access.log logstash_json; #定义日志路径为/var/log/nginx/json.access.log,并引用在主配置文件nginx.conf中定义的json日志格式 include /etc/nginx/default.d/*.conf; location / { root /usr/share/nginx/html; index index.html index.htm; } error_page 404 /404.html; location = /404.html { root /usr/share/nginx/html; } error_page 500 502 503 504 /50x.html; location = /50x.html { root /usr/share/nginx/html; } }
4、重启nginx，查看日志格式是json格式了：
[root@node5 ~]# tail /var/log/nginx/json.access.log {"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"} {"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"} {"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"} {"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"} {"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.001,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"} {"@timestamp":"2016-04-12T22:15:19+08:00","host":"192.168.10.205","clientip":"192.168.10.205","size":3698,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"192.168.10.205","url":"/index.html","domain":"192.168.10.205","xff":"-","referer":"-","agent":"ApacheBench/2.3","status":"200"}
5、在线效验日志格式是否正确：
效验地址：http://www.bejson.com/
五：画图功能
在地图显示IP的访问次数统计：
1、在elasticsearch服务器用户家目录下载一个Filebeat 模板：
cd ~ curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json #这是一个模板文件
2、加载模板：
[root@elk-server1 ~]# curl -XPUT 'http://192.168.0.251:9200/_template/filebeat?pretty' -d@filebeat-index-template.json #是elasticsearch监听的IP地址 { "acknowledged" : true #一定要返回true才表示成功 }
3、下载GeoIP 数据库文件：
[root@elk-server1 ~]# cd /etc/logstash/ [root@elk-server1 logstash]# curl -O "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz" [root@elk-server1 logstash]# gunzip GeoLiteCity.dat.gz [root@elk-server1 logstash]# ls conf.d GeoLiteCity.dat #确认文件存在
4、配置logstash使用GeoIP：
[root@elk-server1 logstash]# vim /etc/logstash/conf.d/11-mobile-tomcat-access.conf #logstash的文件配置要以.conf结尾
input { redis { data_type => "list" key => "mobile-tomcat-access-log" host => "192.168.0.251" port => "6379" db => "0" codec => "json" } } #input部分为从redis读取客户端logstash分析提交后的访问日志 filter { if [type] == "mobile-tomcat" { geoip { source => "client" #client 是客户端logstash收集日志时定义的公网IP的key名称，一定要和实际名称一致，因为要通过此名称获取到其对于的ip地址 target => "geoip" database => "/etc/logstash/GeoLiteCity.dat" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]", "float"] } } } output { if [type] == "mobile-tomcat" { elasticsearch { hosts => ["192.168.0.251"] manage_template => true index => "logstash-mobile-tomcat-access-log-%{+YYYY.MM.dd}" #index的名称一定要是logstash开头的，否则会在使用地图的时候出现geoIP type无法找找到的类似错误 flush_size => 2000 idle_flush_time => 10 } } }
5、在kibana界面添加新的索引，然后visualize---->Tile map---->From a new search---->Select a index patterm--->选择之前的index---->Geo coordinates，然后点绿色的运行按钮即可：
