Logstash收集Nginx、tomcat、java，TCP日志
# 二、部署Logstash
## 2.1 logstash环境准备及安装
Logstash是一个开源的数据收集引擎，可以水平伸缩，而且logstash整个ELK当中拥有最多插件的一个组件，其可以接受来自不同源的数据并同意输出到指定的且可以使多个不同目的地
### 2.1.1 环境准备
关闭防火墙和selinux，并且安装java环境（参照上文）
### 2.1.2 安装logstash
```
[root@study63 src]# yum install logstash-5.4.0.rpm
[root@study63 src]# rpm -qpl logstash-5.4.0.rpm | less
```
## 2.2 测试logstash
### 2.2.1 测试标准输入和输出
```
[root@study63 src]# /usr/share/logstash/bin/logstash -e '{}'
```
```
[root@study63 src]# /usr/share/logstash/bin/logstash -e '{}'
hello
The stdin plugin is now waiting for input:
15:25:19.174 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
{
"@timestamp" => 2020-04-07T07:25:19.355Z, #当前时间的发生时间
"@version" => "1", #事件的版本号，一个事件就是一个ruby对象
"host" => "study63", #标记事件发生在哪里
"message" => "hello" #消息的具体内容
```
### 2.2.2 测试输出到文件
```
[root@study63 src]# /usr/share/logstash/bin/logstash -e 'input{ stdin{} } output{ file { path => "/tmp/log-%{+YYYY.MM.dd}.log" gzip => true} }'
[root@study63 tmp]# tail test-2020.04.07.log
```
### 2.2.3测试输出到elasticsearch
```
/usr/share/logstash/bin/logstash -e 'input{ stdin{} } output{ elasticsearch {hosts => ["10.0.0.62:9200","10.0.0.63:9200"] index => "logstash-test-%{+YYYY.MM.dd}" }}'
```
# 三、Kibana部署即日志收集
## 3.1 安装并配置kibana
### 3.1.1 rpm方式
```
[root@study62 src]# yum install kibana-5.4.0-x86_64.rpm -y
```
```
[root@study62 src]# vim /etc/kibana/kibana.yml
[root@study62 src]# grep "^[a-Z]" /etc/kibana/kibana.yml
server.port: 5601
server.host: "10.0.0.62"
elasticsearch.url: "http://10.0.0.62:9200"
```
### 3.1.2 启动kibana服务并验证
```
[root@study62 src]# systemctl start kibana && systemctl enable kibana
[root@study62 src]# ss -tln
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:9100 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 10.0.0.62:5601 *:*
LISTEN 0 128 ::ffff:10.0.0.62:9200 :::*
LISTEN 0 128 ::ffff:10.0.0.62:9300 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 100 ::1:25 :::*
```
### 3.1.3 浏览器访问http://10.0.0.62:5601
## 3.2 系统日志收集案例
### 3.2.1 logstash配置文件
```
[root@study62 src]# vim /etc/logstash/conf.d/system.conf
input {
file {
path => "/var/log/messages"
type => "systemlog"
start_position => "beginning"
stat_interval => "2"
}
}
output{
elasticsearch {
hosts => ["10.0.0.62:9200","10.0.0.63:9200"]
index => "logstash-systemlog-%{+YYYY.MM.dd}"
}
}
```
### 3.2.2 检查配置文件是否有误
```
[root@study62 src]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf -t
```
### 3.2.3 修改messages文件权限，logstash具有读权限
```
[root@study62 ~]# chmod 644 /var/log/messages
```
### 3.2.4 重启服务并在kibana中添加索引
```
[root@study62 src]# systemctl restart logstash.service
```
## 3.3 if判断多个type类型
```
[root@study62 ~]# vi /etc/logstash/conf.d/system.conf
input {
file {
path => "/var/log/messages"
type => "systemlog"
start_position => "beginning"
stat_interval => "2"
}
file {
path => "/var/log/lastlog"
type => "system-last"
start_position => "beginning"
stat_interval => "2"
}
}
output{
if [type] == "systemlog" {
elasticsearch {
hosts => ["10.0.0.62:9200"]
index => "logstash-systemlog-%{+YYYY.MM.dd}"
}
file {
path => "/tmp/last.log"
}
}
if [type] == "system-last" {
elasticsearch {
hosts => ["10.0.0.63:9200"]
index => "logstash-lastlog-%{+YYYY.MM.dd}"
}
}
}
[root@study62 src]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf -t
```
## 3.4 收集Nginx的json格式日志
### 3.4.1 安装并配置nginx
```
[root@study62 ~]# yum install nginx
[root@study62 ~]# vi /etc/nginx/nginx.conf
location /nginxweb {
root html;
index index.html index.htm;
}
[root@study62 ~]# cd /usr/share/nginx/html/
[root@study62 ~]# mkdir nginxweb
[root@study62 ~]# cd nginxweb
[root@study62 ~]# echo "Nginx Web" > index.html
```
### 3.4.2 启动nginx并访问web页面
```
[root@study62 ~]# systemctl start nginx
[root@study62 ~]# ss -tnl
```
### 3.4.3 将nginx日志转换为json格式
```
[root@study62 ~]# vim /etc/nginx/nginx.conf
log_format access_json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"size": $body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status"}';
access_log /var/log/nginx/access.log access_json;
[root@study62 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@study62 ~]# systemctl restart nginx
```
### 3.4.5 使用ab来填充日志数据
```
[root@study62 ~]# yum install httpd-tools -y
[root@study62 ~]# ab -n1000 -c100 http://10.0.0.62/nginxweb/index.html
```
### 3.4.6 Python脚本
```
[root@study62 ~]# cat log.py
#!/usr/bin/env python
data = {"@timestamp":"2020-04-08T18:51:06+08:00","host":"10.0.0.62","cLientip":"10.0.0.62","size": 10,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.62","url":"/nginxweb/index.html","domain":"10.0.0.62","xff":"-","referer":"-","status":"200"}
ip = data.get("cLientip")
print ip
```
### 3.4.6 添加logstash配置文件
```
[root@study62 ~]# vim /etc/logstash/conf.d/nginx-accesslog.conf
input {
file {
path => "/var/log/nginx/access.log"
type => "nginx-access-log"
start_position => "beginning"
stat_interval => "2"
}
}
output {
elasticsearch {
hosts => ["10.0.0.62:9200"]
index => "logstash-nginx-access-log-%{+YYYY.MM.dd}"
}
}
[root@study62 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-accesslog.conf -t
[root@study62 ~]# systemctl restart logstash.service
```
### 3.4.7 kibana中添加索引
## 3.5 Logstash收集Tomcat访问日志
### 3.5.1 安装并配置Tomcat
#### 3.5.1.1 安装tomcat
```
[root@study63 src]# ll
total 363464
-rw-r--r-- 1 root root 10312541 Apr 13 08:30 apache-tomcat-8.5.54.tar.gz
[root@study63 src]# tar xf apache-tomcat-8.5.54.tar.gz -C /opt/
[root@study63 src]# cd /opt/
[root@study63 opt]# ls
apache-tomcat-8.5.54 src
[root@study63 opt]# ln -sv /opt/apache-tomcat-8.5.54 /opt/tomcat
‘/opt/tomcat’ -> ‘/opt/apache-tomcat-8.5.54’
```
#### 3.5.1.2 修改日志格式
```
[root@study63 opt]# cd tomcat/
[root@study63 tomcat]# cd webapps/
[root@study63 webapps]# mkdir webdir
[root@study63 webapps]# cd webdir/
[root@study63 webdir]# vim index.html
[root@study63 webdir]# cd ..
[root@study63 webapps]# ../bin/catalina.sh start
Using CATALINA_BASE: /opt/tomcat
Using CATALINA_HOME: /opt/tomcat
Using CATALINA_TMPDIR: /opt/tomcat/temp
Using JRE_HOME: /usr
Using CLASSPATH: /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
Tomcat started.
```
```
[root@study63 tomcat]# vim conf/server.xml
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="tomcat_access_log" suffix=".log"
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
[root@study63 tomcat]# ./bin/catalina.sh stop
Using CATALINA_BASE: /opt/tomcat
Using CATALINA_HOME: /opt/tomcat
Using CATALINA_TMPDIR: /opt/tomcat/temp
Using JRE_HOME: /usr
Using CLASSPATH: /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
[root@study63 tomcat]# ./bin/catalina.sh start
Using CATALINA_BASE: /opt/tomcat
Using CATALINA_HOME: /opt/tomcat
Using CATALINA_TMPDIR: /opt/tomcat/temp
Using JRE_HOME: /usr
Using CLASSPATH: /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
Tomcat started.
[root@study63 tomcat]# tail logs/tomcat_access_log.2020-04-13.log
{"clientip":"10.0.0.1","ClientUser":"-","authenticated":"-","AccessTime":"[13/Apr/2020:08:48:22 +0800]","method":"GET / HTTP/1.1","status":"200","SendBytes":"11215","Query?string":"","partner":"-","AgentVersion":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"}
```
### 3.5.2 编写logstash配置文件
```
[root@study63 tomcat]# vim /etc/logstash/conf.d/tomcat.access.conf
input {
file {
path => "/opt/tomcat/logs/tomcat_access_log.*.log"
type => "tomcat-accesslog"
start_position => "beginning"
stat_interval => "2"
}
}
output {
if [type] == "tomcat-accesslog" {
elasticsearch {
hosts => ["10.0.0.63:9200"]
index => "logstash-tomcat063-accesslog-%{+YYYY.MM.dd}"
}}
}
[root@study63 tomcat]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat.access.conf -t
```
### 3.5.3 以root用户启动logstash并检查es中状态
```
[root@study63 ~]# vi /etc/systemd/system/logstash.service
[Service]
Type=simple
User=root
Group=root
[root@study63 ~]# systemctl daemon-reload
[root@study63 tomcat]# systemctl restart logstash.service
```
### 3.6 收集Java日志
### 3.6.1 多行匹配
```
[root@study62 ~]# /usr/share/logstash/bin/logstash -e 'input { stdin { codec => multiline { pattern => "^\[" negate => true what => "previous" } } } output{ stdout { codec => "rubydebug" } }'
```
### 3.6.2 添加Logstash配置文件
```
[root@study62 ~]# vi /etc/logstash/conf.d/java.conf
input {
file {
path => "/data/logs/elk-cluster.log"
type => "elasticsearch-java-log"
start_position => "beginning"
stat_interval => "2"
codec => multiline
{
pattern => "^\["
negate => true
what => "previous"
}
}
}
output {
if [type] == "elasticsearch-java-log"{
elasticsearch {
hosts => ["10.0.0.62:9200"]
index => "elasticsearch-java-log-%{+YYYY.MM.dd}"
}
}
}
[root@study62 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/java.conf -t
```
### 3.6.3 启动logstash并检查es中状态
```
[root@study62 ~]# systemctl restart logstash.service
```
## 3.7 收集TCP日志
### 3.7.1 添加logstash配置文件
```
[root@study63 ~]# vi /etc/logstash/conf.d/tcp.conf
input {
tcp {
port => 7800
mode => "server"
type => "tcplog"
start_position => "beginning"
stat_interval => "2"
}
}
output {
stdout {
codec => "rubydebug"
}
}
```
### 3.7.2 验证配置
```
[root@study63 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp.conf -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
Configuration OK
13:54:45.560 [LogStash::Runner] INFO logstash.runner - Us
```
### 3.7.3 安装nc工具
```
[root@study63 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp.conf
[root@study62 ~]# yum install nc -y
[root@study62 ~]# echo "tcpdata" | nc 10.0.0.63 5600
```
### 3.7.4 配置多个input/output案例
```
[root@study63 ~]# mv /etc/logstash/conf.d/tcp.conf /opt/
[root@study63 ~]# vi /etc/logstash/conf.d/tomcat-access.conf
input {
file {
path => "/opt/tomcat/logs/tomcat_access_log.*.log"
type => "tomcat-accesslog"
start_position => "beginning"
stat_interval => "2"
}
tcp {
port => 5600
mode => "server"
type => "tcplog"
start_position => "beginning"
stat_interval => "2"
}
}
output {
if [type] == "tomcat-accesslog" {
elasticsearch {
hosts => ["10.0.0.63:9200"]
index => "logstash-tomcat063-accesslog-%{+YYYY.MM.dd}"
}}
if [type] == "tcplog" {
elasticsearch {
hosts => ["10.0.0.63:9200"]
index => "tcp-063-%{+YYYY.MM.dd}"
}}
}
[root@study63 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tomcat-access.conf -t
```
#### 3.7.4.1 添加验证数据并在ES中验证
```
[root@study62 ~]# nc 10.0.0.63 5600 < /etc/passwd
```
```
[root@study62 ~]# echo "伪设备1" > /dev/tcp/10.0.0.63/7800
```
