linux系统ElK基础(2)

一、Logstash收集日志

1.Logstash的配置文件

[root@web01 ~]# vim /etc/logstash/logstash.yml
path.config: /etc/logstash/conf.d

2.logstash收集日志文件到文件

[root@web01 ~]# vim /etc/logstash/conf.d/file_file.conf
input {
  file {
    path => "/var/log/messages"
    start_position => "beginning"
  }
}
output {
  file {
    path => "/tmp/messages_%{+YYYY-MM-dd}.log"
  }
}

3.logstash收集日志文件到ES

[root@web01 ~]# vim /etc/logstash/conf.d/file_es.conf
input {
  file {
    path => "/var/log/messages"
    start_position => "beginning"
  }
}
output {
  elasticsearch {
    hosts => ["172.16.1.51:9200"]
    index => "messages_%{+YYYY-MM-dd}.log"
  }
}

4.Logstash收集多日志到文件

[root@web01 ~]# vim /etc/logstash/conf.d/file_file.conf
input {
  file {
    type => "messages_log"
    path => "/var/log/messages"
    start_position => "beginning"
  }
  file {
    type => "secure_log"
    path => "/var/log/secure"
    start_position => "beginning"
  }       
}        
output {  
  if [type] == "messages_log" { 
    file {
      path => "/tmp/messages_%{+YYYY-MM-dd}"
    }        
  }
  if [type] == "secure_log" {
    file {
      path => "/tmp/secure_%{+YYYY-MM-dd}"
    }
  } 
}

5.Logstash收集多日志到ES

1）方法一：

[root@web01 ~]# vim /etc/logstash/conf.d/more_es.conf 
input {
  file {
    type => "messages_log"
    path => "/var/log/messages"
    start_position => "beginning"
  }
  file {
    type => "secure_log"
    path => "/var/log/secure"
    start_position => "beginning"
  }
}
output {
  if [type] == "messages_log" {
    elasticsearch {
      hosts => ["10.0.0.51:9200"]
      index => "messages_%{+YYYY-MM-dd}"
    }
  }
  if [type] == "secure_log" {
    elasticsearch {
      hosts => ["10.0.0.51:9200"]
      index => "secure_%{+YYYY-MM-dd}"
    }
  }
}

[root@web01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/more_es.conf &

#启动后查看页面

2）方法二：

[root@web01 ~]# vim /etc/logstash/conf.d/more_es_2.conf 
input {
  file {
    type => "messages_log"
    path => "/var/log/messages"
    start_position => "beginning"
  }
  file {
    type => "secure_log"
    path => "/var/log/secure"
    start_position => "beginning"
  }
}
output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "%{type}_%{+YYYY-MM-dd}"
  }
}

[root@web01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/more_es_2.conf --path.data=/data/logstash/more_es_2 &

3）启动多实例

#创建不同的数据目录
[root@web01 ~]# mkdir /data/logstash/more_es_2
[root@web01 ~]# mkdir /data/logstash/more_es

#启动时使用--path.data指定数据目录
[root@web01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/more_es.conf --path.data=/data/logstash/more_es &
[root@web01 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/more_es_2.conf --path.data=/data/logstash/more_es_2 &

#如果资源充足，可以使用多实例收集多日志，如果服务器资源不足，启动不了多实例，配置一个文件收集多日志启动

二、Logstash收集Tomcat日志

1.安装Tomcat

1.安装java环境
[root@web01 ~]# rpm -ivh jdk-8u181-linux-x64.rpm

2.上传包
[root@web01 ~]# rz apache-tomcat-10.0.0-M7.tar.gz

3.解压
[root@web01 ~]# tar xf apache-tomcat-10.0.0-M7.tar.gz -C /usr/local/

4.做软连接
[root@web01 ~]# ln -s /usr/local/apache-tomcat-10.0.0-M7 /usr/local/tomcat

5.启动Tomcat
[root@web01 ~]# /usr/local/tomcat/bin/startup.sh

6.访问页面 10.0.0.7:8080

2.配置Logstash收集Tomcat日志到文件

[root@web01 ~]# vim /etc/logstash/conf.d/tomcat_file.conf
input {
  file {
    path => "/usr/local/tomcat/logs/localhost_access_log.*.txt"
    start_position => "beginning"
  }
}
output {
  file {
    path => "/tmp/tomcat_%{+YYYY-MM-dd}.log"
  }
}

3.配置Logstash收集Tomcat日志到ES

[root@web01 ~]# vim /etc/logstash/conf.d/tomcat_es.conf
input {
  file {
    path => "/usr/local/tomcat/logs/localhost_access_log.*.txt"
    start_position => "beginning"
  }
}
output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "tomcat_%{+YYYY-MM-dd}.log"
  }
}

三、收集Tomcat日志修改格式

#收集tomcat日志，当遇到报错时，一条报错会被分割成很多条数据，不方便查看

解决方法：
1.修改tomcat日志格式为json
	1）开发修改输出日志为json
	2）修改tomcat配置，日志格式为json
2.使用logstash的input插件下的mutiline模块

1.方法一：修改tomcat日志格式

1）配置tomcat日志为json格式

[root@web01 ~]# vim /usr/local/tomcat/conf/server.xml
#把原来的日志格式注释，添加我们的格式
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="tomcat_access_json" suffix=".log"
               pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&quot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quot;:&quot;%{User-Agent}i&quot;}"/>

2）重启tomcat

[root@web01 ~]# /usr/local/tomcat/bin/shutdown.sh
[root@web01 ~]# /usr/local/tomcat/bin/startup.sh

3）配置收集新的tomcat日志

[root@web01 ~]# vim /etc/logstash/conf.d/tomcat_json_es.conf
input {
  file {
    path => "/usr/local/tomcat/logs/tomcat_access_json.*.log"
    start_position => "beginning"
  }
}
output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "tomcat_json_%{+YYYY-MM-dd}.log"
  }
}

2.方法二：使用mutiline模块收集日志

1）配置收集日志测试

[root@web01 ~]# vim /etc/logstash/conf.d/test_mutiline.conf
input {
  stdin {
    codec => multiline {
	  #以[开头
      pattern => "^\["
      #匹配到
      negate => true
      #向上合并，向下合并是next
      what => "previous"
    }
  }
}
output {
  stdout {
    codec => json
  }
}

#测试，输入内容不会直接输出，当遇到以 [ 开头才会收集以上的日志

2）配置收集tomcat错误日志

[root@web01 ~]# vim /etc/logstash/conf.d/tomcat_mutiline.conf 
input {
  file {
    path => "/usr/local/tomcat/logs/tomcat_access_json.*.log"
    start_position => "beginning"
    codec => multiline {
      pattern => "^\["
      negate => true
      what => "previous"
    }
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "tomcat_json_%{+YYYY-MM-dd}"
    codec => "json"
  }
}

3）将错误日志写入

[root@web01 ~]# cat 1.txt >> /usr/local/tomcat/logs/tomcat_access_json.2020-08-14.log

4）页面查看数据

![img](file:///C:\Users\oldboy\AppData\Roaming\Tencent\Users\1240206455\QQ\WinTemp\RichOle\AM25IS`()KVQFI@Y5NOD5C9.png)

四、收集Nginx日志

1.安装Nginx

[root@web01 ~]# yum install -y nginx

2.配置Nginx日志格式

[root@web01 ~]# vim /etc/nginx/nginx.conf
... ...
http {
    log_format  json  '{"@timestamp":"$time_iso8601",'
                      '"host":"$server_addr",'
                      '"clientip":"$remote_addr",'
                      '"size":$body_bytes_sent,'
                      '"responsetime":$request_time,'
                      '"upstreamtime":"$upstream_response_time",'
                      '"upstreamhost":"$upstream_addr",'
                      '"http_host":"$host",'
                      '"url":"$uri",'
                      '"referer":"$http_referer",'
                      '"agent":"$http_user_agent",'
                      '"status":"$status"}';

    access_log  /var/log/nginx/access.log  json;
... ...

3.配置收集Nginx日志

[root@web01 ~]# vim /etc/logstash/conf.d/nginx_json.conf
input {
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
  }
}
output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "nginx_json_%{+YYYY-MM-dd}.log"
  }
}

五、获取的日志参数分离

1.方法一：

1）修改tomcat日志收集配置

[root@web01 ~]# vim /etc/logstash/conf.d/tomcat_json_es.conf

input {
  file {
    path => "/usr/local/tomcat/logs/tomcat_access_json.*.log"
    start_position => "beginning"
  }
}

#把收集到的数据进行处理
filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "tomcat_json_%{+YYYY-MM-dd}.log"
  }
}

2）去掉多余数据

#message数据已经拆分，数据还在，去掉message数据
filter {
  json {
    source => "message"
    remove_field => ["message"]
  }
}

2.方法二：

1）修改收集Nginx日志的配置

#nginx不需要配置修改获取日志，只需要收集同时修改格式即可
[root@web01 ~]# vim /etc/logstash/conf.d/nginx_json.conf 
input {
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
    codec => "json"
  }
}
output {
  elasticsearch {
    hosts => ["10.0.0.51:9200"]
    index => "nginx_json_%{+YYYY-MM-dd}.log"
  }
}

六、Logstash收集日志写入redis

1.安装redis

2.配置将数据写入redis

[root@web01 ~]# vim /etc/logstash/conf.d/nginx_to_redis.conf
input {
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
    codec => "json"
  }
}
output {
  redis {
    host => "172.16.1.51"
    port => "6379"
    data_type => "list"
    db => "0"
    key => "nginx_log"
  }
}

posted @ 2020-08-19 14:31 王顺子阅读(148) 评论(0) 编辑收藏举报

刷新页面返回顶部

linux系统ElK基础(2)

一、Logstash收集日志

1.Logstash的配置文件

2.logstash收集日志文件到文件

3.logstash收集日志文件到ES

4.Logstash收集多日志到文件

5.Logstash收集多日志到ES

1）方法一：

2）方法二：

3）启动多实例

二、Logstash收集Tomcat日志

1.安装Tomcat

2.配置Logstash收集Tomcat日志到文件

3.配置Logstash收集Tomcat日志到ES

三、收集Tomcat日志修改格式

1.方法一：修改tomcat日志格式

1）配置tomcat日志为json格式

2）重启tomcat

3）配置收集新的tomcat日志

2.方法二：使用mutiline模块收集日志

1）配置收集日志测试

2）配置收集tomcat错误日志

3）将错误日志写入

4）页面查看数据

四、收集Nginx日志

1.安装Nginx

2.配置Nginx日志格式

3.配置收集Nginx日志

五、获取的日志参数分离

1.方法一：

1）修改tomcat日志收集配置

2）去掉多余数据

2.方法二：

1）修改收集Nginx日志的配置

六、Logstash收集日志写入redis

1.安装redis

2.配置将数据写入redis

公告