ELK-logstash日志收集

前提需要 logstash 用户对被收集的日志文件有读的权限并对写入的文件有写权限。如果启动用户是logstash。

默认启动加载配置文件：/etc/logstash/conf.d/，所有的日志收集配置文件应该放在这个路径。

1、 收集 单个 系统 日志 并 输出至文件

[root@linux-host3 ~]# cat /etc/logstash/conf.d/system-log.conf
input {
  file {
  type => "messagelog"
  path => "/var/log/messages"
  start_position => "beginning" #第一次从头收集，之后从新添加的日志收集
  }
}

output {
  file {
  path => "/tmp/%{type}.%{+yyyy.MM.dd}"
  }
}

 检测 配置文件语法是否 正确

/usr/share/logstash/bin/logstash  -f   /etc/logstash/conf.d/syslog.conf –t

 生成数据 并 验证

[root@linux-host3 ~]# echo "test" >> /var/log/messages
[root@linux-host3 ~]# tail /tmp/messagelog.2017.04.20 #验证是否生成文件
{"path":"/var/log/messages","@timestamp":"2017-04-20T07:12:16.001Z","@version":"1","host":"linux-host3.exmaple.com","message":"test","type":"messagelog"}   # 可以看到logstash默认添加了key，包含主机host

 2、通过 logstash  收集多个日志文件

[root@linux-host3 logstash]# cat /etc/logstash/conf.d/system-log.conf
input {
  file {
    path => "/var/log/messages" #日志路径
    type => "systemlog" #事件的唯一类型
    start_position => "beginning" #第一次收集日志的位置
    stat_interval => "3" #日志收集的间隔时间
  }

  file {
    path => "/var/log/secure"
    type => "securelog"
    start_position => "beginning"
    stat_interval => "3"
  }
}

output {
  if [type] == "systemlog" {
    elasticsearch {
      hosts => ["192.168.15.11:9200"]     # es集群
      index => "system-log-%{+YYYY.MM.dd}"   # index，dd指按天创建index
    }
  }

if [type] == "securelog" {
  elasticsearch {
    hosts => ["192.168.15.11:9200"]
    index => "secury-log-%{+YYYY.MM.dd}"
    }
  }
}

重启 logstash  并查看日志是否有报错，加载配置

[root@linux-host3 ~]# chmod 644 /var/log/secure
[root@linux-host3 ~]# chmod 644 /var/log/messages
[root@linux-host3 logstash]# systemctl restart logstash

测试

[root@linux-host3 logstash]# echo "test" >> /var/log/secure
[root@linux-host3 logstash]# echo "test" >> /var/log/messages

 3. 通过 logtsash  收集 tomcat 和 和 java  日志

tomcat  日志转 json

[root@linux-host6 tomcat]# vim conf/server.xml
<Valve  className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="tomcat_access_log" suffix=".log"
pattern="{&quot;clientip&quot;:&quot;%h&quot;,&quot;ClientUser&quot;:&quot;%l
&quot;,&quot;authenticated&quot;:&quot;%u&quot;,&quot;AccessTime&quot;:&qu
ot;%t&quot;,&quot;method&quot;:&quot;%r&quot;,&quot;status&quot;:&quot;%s&
quot;,&quot;SendBytes&quot;:&quot;%b&quot;,&quot;Query?string&quot;:&quot;
%q&quot;,&quot;partner&quot;:&quot;%{Referer}i&quot;,&quot;AgentVersion&quo
t;:&quot;%{User-Agent}i&quot;}"/>

验证日志格式

 在 tomcat  服务器安装 logstash  收集 tomcat 和系统 日志

[root@linux-host6 ~]# cat /etc/logstash/conf.d/tomcat-es.conf
input {
  file {
    path => "/usr/local/tomcat/logs/localhost_access_log.*.txt"
    start_position => "end"
    type => "tomct-access-log"
  }

  file {
    path => "/var/log/messages"
    start_position => "end"
    type => "system-log"
  }
}

output {
  if [type] == "tomct-access-log" {
    elasticsearch {
      hosts => ["192.168.15.11:9200"]
      index => "logstash-tomcat-1516-access-%{+YYYY.MM.dd}"
       codec => "json"
    }
  }

if [type] == "system-log" {
  elasticsearch {
    hosts => ["192.168.15.12:9200"] #写入到不同的 ES 服务器
    index => "system-log-1516-%{+YYYY.MM.dd}"
    }
  }
}

修改权限

[root@linux-host6 ~]# chmod 644 /var/log/messages

 访问 head  插件 验证索引