ELK之三-----JAVA日志多行合并、tomcat json日志收集与处理
实战一：通过Logstash收集tomcat服务器的日志
1、配置JDK环境
1、解压JDK包，创建软链接
[root@tomcat-web1 src]# tar xvf jdk-8u212-linux-x64.tar.gz [root@tomcat-web1 src]# ln -sv /usr/local/src/jdk1.8.0_212/ /usr/local/jdk ‘/usr/local/jdk/jdk1.8.0_212’ -> ‘/usr/local/src/jdk1.8.0_212/’ [root@tomcat-web1 src]# ln -sv /usr/local/jdk/bin/java /usr/bin
2、配置java的环境变量
[root@tomcat-web1 ~]# vim /etc/profile.d/jdk.sh #配置环境变量 export HISTTIMEFORMAT="%F %T `whoami`" export export LANG="en_US.utf-8" export JAVA_HOME=/usr/local/jdk export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$PATH:$JAVA_HOME/bin [root@tomcat-web1 ~]# . /etc/profile.d/jdk.sh #将环境变量生效
3、查看版本信息及java家目录信息
[root@tomcat-web1 src]# java -version java version "1.8.0_212" Java(TM) SE Runtime Environment (build 1.8.0_212-b10) Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode) [root@tomcat-web1 src]# echo $JAVA_HOME /usr/local/jdk
2、配置tomcat服务并启动
1、解压tomcat服务器的安装包，并创建tomcat软链接
[root@tomcat-web1 ~]# mkdir /apps [root@tomcat-web1 ~]# cd /apps/ [root@tomcat-web1 apps]# ls apache-tomcat-8.5.42 apache-tomcat-8.5.42.tar.gz tomcat [root@tomcat-web1 apps]# tar xvf apache-tomcat-8.5.42.tar.gz [root@tomcat-web1 apps]# ln -s /apps/apache-tomcat-8.5.42 /apps/tomcat #创建tomcat软链接
2、启动tomcat服务
[root@tomcat-web1 apps]# /apps/tomcat/bin/startup.sh Using CATALINA_BASE: /apps/tomcat Using CATALINA_HOME: /apps/tomcat Using CATALINA_TMPDIR: /apps/tomcat/temp Using JRE_HOME: /usr/local/jdk Using CLASSPATH: /apps/tomcat/bin/bootstrap.jar:/apps/tomcat/bin/tomcat-juli.jar Tomcat started.
3、设置访问网站路径
[root@tomcat-web1 apps]# vim /apps/tomcat/conf/server.xml <Host name="localhost" appBase="/data/tomcat/tomcat_webdir"
4、查看tomcat访问页面，此访问页面是前面自制的页面，能访问，说明tomcat服务正常。
5、修改tomcat服务的配置文件，收集log日志为json格式：/apps/tomcat/bin/server.xml，实际的tomcat访问log日志文件目录在：/apps/tomcat/logs/tomcat_access_log.2020-03-13.log
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="tomcat_access_log" suffix=".log" # 将log日志文件名进行修改，以.log为后缀的文件
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u",&quo #修改为json格式日志文件。
t;AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","
Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
访问此时的tomcat日志文件：/apps/tomcat/logs/tomcat_access_log.2020-03-13.log ，验证日志是否是json格式的，可以在网上查看。
[root@logstash ~]# tail -f /apps/tomcat/logs/tomcat_access_log.2020-03-13.log
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:19:16 +0800]","method":"GET / HTTP/1.1","status":"404","SendBytes":"1078","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:19:16 +0800]","method":"GET /favicon.ico HTTP/1.1","status":"404","SendBytes":"1078","Query?string":"","partner":"http://192.168.7.102:8080/","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:22:09 +0800]","method":"GET / HTTP/1.1","status":"404","SendBytes":"1078","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:22:38 +0800]","method":"GET /myapp HTTP/1.1","status":"302","SendBytes":"-","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
{"clientip":"192.168.7.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Mar/2020:15:22:38 +0800]","method":"GET /myapp/ HTTP/1.1","status":"200","SendBytes":"14","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"}
3、收集tomcat日志文件
需要将logstash主机启动脚本改为root启动用户，否则无法收集到tomcat的日志文件
[root@logstash conf.d]# vim /etc/systemd/system/logstash.service User=root Group=root
重启logstash服务
[root@logstash conf.d]# systemctl restart logstash
创建/etc/logstash/conf.d目录下的配置文件：tomcat-java-log.conf
input {
file {
path => "/var/log/logstash/logstash-plain.log" #收集java的日志文件目录
start_position => "beginning"
stat_interval => 3
type => "java-log"
}
file {
path => "/apps/tomcat/logs/tomcat_access_log.*.log" # 收集tomcat日志的文件目录
start_position => "beginning"
stat_interval => 3
type => "tomcat-access-log"
codec => "json" # 输出tomcat 的json日志格式
}
}
output {
if [type] == "java-log" {
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "javalog-7-102-%{+YYYY.MM.dd}"
}
}
if [type] == "tomcat-access-log" {
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "tomcat-access-log-7-102-%{+YYYY.MM.dd}"
}
}
}
在kibana网站上创建tomcat日志的索引
在discover选项中，查看添加后的tomcat日志文件
3、收集java日志json格式的文件
1、在/etc/logstash/conf.d/目录下创建一个java.conf文件，修改此配置文件，将logstash的日志文件收集到elasticsearch主机上。
input {
file {
path => "/var/log/logstash/logstash-plain.log"
start_position => "beginning"
stat_interval => 3
type => "java-log"
}
}
output {
if [type] == "java-log" {
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "javalog-7-102-%{+YYYY.MM.dd}"
}
}
}
2、重启logstash服务，并观察logstash启动情况
# systemctl restart logstash
在/var/log/logstash/logstash-plain.log文件中可以查看此时的logstash服务器启动情况。
[root@logstash conf.d]# tail -f /var/log/logstash/logstash-plain.log
[2020-03-13T15:49:04,872][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.7.100:9200"]}
[2020-03-13T15:49:11,231][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_7d5605c109b000fd1e6e680ae503330d", :path=>["/var/log/logstash/logstash-plain.log"]}
[2020-03-13T15:49:11,291][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_452905a167cf4509fd08acb964fdb20c", :path=>["/var/log/messages"]}
[2020-03-13T15:49:11,297][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_d883144359d3b4f516b37dba51fab2a2", :path=>["/var/log/nginx/access.log"]}
[2020-03-13T15:49:11,387][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x35cee74 run>"}
[2020-03-13T15:49:11,453][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2020-03-13T15:49:11,456][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2020-03-13T15:49:11,478][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2020-03-13T15:49:11,622][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-03-13T15:49:12,874][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} # 查看到此信息，说明logstash已经启动了。
3、在kibana控制台添加java日志文件
4、此时在discover选项可以看到添加的javalog日志。
5、日志多行合并处理—multiline插件（重点）
目前5.5版本支持此插件，6.x版本已经不需要此版本，可以自动合并日志文件。
官方文档：https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html
介绍multiline
pattern：正则匹配从哪行合并
negate：true/false，匹配到pattern 部分开始合并，还是不配到的合并
input {
file {
path => "/var/log/logstash/logstash-plain.log" # 要采集的log日志
start_position => "beginning"
codec => multiline {
pattern => "^\[" # 以[开头开始匹配
negate => true
what => "previous"
}
}
}
output { # 输出到elasticsearch主机上
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "logstash-log-7-100-%{+YYYY.MM.dd}"
}}
