Logstash深入收集Java日志

Logstash深入收集Java日志

没有修改Json格式

在企业中，我们看到tomcat日志遇到异常(exception)一条日志可能是几行或者十几行甚至几十行，组成的，那么，我们需要将多行日志变成一行日志，来收集。

这里我们有几种方式可以实现：
1.将日志改成Json格式
在企业中，想要将java日志改成json格式，并没有那么容易。
格式不是你想改，想改就能改，让我挣开，让我明白，放手你的爱~~~~
因为将日志改成Json格式，查看起来会很难受，有些开发人员不希望将日志格式改成Json的，所以，在改日志格式之前需要跟开发人员进行沟通，那么将tomcat日志格式改成Json格式也有两种方式。
1）开发自己更改，通过程序代码，或者log4j
2）运维修改tomcat的server配置文件

准备tomcat环境

# 1.安装tomcat
[root@elkstack03 ~]# yum install -y tomcat

# 2.部署tomcat代码
[root@elkstack03 ~]# vim /usr/share/tomcat/webapps/ROOT/index.jsp
test tomcat

# 3.启动tomcat
[root@elkstack03 ~]# systemctl start tomcat

使用Logstash收集java日志

[root@elkstack03 tomcat]# vim /etc/logstash/conf.d/tomcat_file_es.conf
input{
        file{
                type => "tomcat_access_log"
                path => "/var/log/tomcat/localhost_access_log.2022-09-07.txt"
                start_position => "beginning"
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
        }
}


[root@elkstack03 tomcat]# /usr/share/logstash/bin/logstash --path.data=/var/lib/logstash/tomcat_file_es -f /etc/logstash/conf.d/tomcat_file_es.conf &

修改tomcat日志格式为Json

[root@elkstack03 tomcat]# vim /etc/tomcat/server.xml
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="tomcat_access_log" suffix=".log"
               pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/> 
               
               
               
               
               
137         <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
138                prefix="localhost_access_log." suffix=".txt"
139                pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;
    %u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&
    quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&
    quot;%{User-Agent}i&quot;}" />

使用Logstash来解析json格式

vim tomcat_file_es_json.conf 
input{
        file{
                type => "tomcat_access_log_json"
                path => "/var/log/tomcat/localhost_access_log.*.txt"
                start_position => "end"
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
                codec => "json"
        }
}

[root@elkstack03 tomcat]# /usr/share/logstash/bin/logstash --path.data=/var/lib/logstash/tomcat_access_json -f /etc/logstash/conf.d/tomcat_file_es_json.conf &

还是一坨

解析Json格式

[root@elkstack03 conf.d]# vim tomcat_file_es_json.conf 
input{
        file{
                type => "tomcat_access_log_json"
                path => "/var/log/tomcat/localhost_access_log.*.txt"
                start_position => "end"
        }
}

filter{
        json{
                source => "message"
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
                codec => "json"
        }
}

删除多余的message字段

[root@elkstack03 conf.d]# vim tomcat_file_es_json.conf 
input{
        file{
                type => "tomcat_access_log_json"
                path => "/var/log/tomcat/localhost_access_log.*.txt"
                start_position => "end"
        }
}

filter{
        json{   
                source => "message"
                remove_field => ["message"]
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
                codec => "json"
        }
}

[root@elkstack03 conf.d]#  /usr/share/logstash/bin/logstash --path.data=/var/lib/logstash/tomcat_access_json -f /etc/logstash/conf.d/tomcat_file_es_json.conf &

Logstash收集catlina日志（异常错误日志）

[root@elkstack03 conf.d]# vim /etc/logstash/conf.d/catlina_file_es.conf
input{
        file{
                type => "tomcat_catlina_log"
                path => "/var/log/tomcat/catalina.*.log"
                start_position => "beginning"
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
        }
}

[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash --path.data=/var/lib/logstash/tomcat_catlina/ -f /etc/logstash/conf.d/catlina_file_es.conf &

logstash多行合并

[root@elkstack03 conf.d]# vim /etc/logstash/conf.d/catlina_file_es.conf 
input{
        file{
                type => "tomcat_catlina_log"
                path => "/var/log/tomcat/catalina.*.log"
                start_position => "beginning"
                codec => multiline {
                    pattern => "^[A-Z]"
                    negate => true
                    what => "previous"
                }
        }
}

output{
        elasticsearch{
                hosts => ["10.0.0.81:9200"]
                index => "%{type}-%{+yyyy.MM.dd}"
        }
}