部署logstash

一 简介：

    Logstash是一个开源的数据收集引擎，可以水平伸缩，而且logstash整个ELK当中拥有最多插件的一个组件，其可以接收来自不同来源的数据并统一输出到指定的且可以是多个不同目的地。

二 安装logstash

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.rpm
yum -y install java-1.8.0-openjdk-devel
yum -y install logstash-6.3.2.rpm 
chown -R logstash. /usr/share/logstash/data/queue

三 测试logstash

    3.1.1测试标准输入和输出：

[root@centos7-1 ~]# /usr/share/logstash/bin/logstash   -e 'input {  stdin{} } output { stdout{  codec => rubydebug }}'
hello word 
{
      "@version" => "1",                         #事件版本号，一个事件就是一个ruby对象
    "@timestamp" => 2018-08-04T03:43:54.901Z,    #当前事件的发生时间
          "host" => "centos7-1",                 #标记事件发生在哪里
       "message" => "hello word"                 #消息的具体内容
}

   3.1.2 测试输出到文件

[root@centos7-1 ~]# /usr/share/logstash/bin/logstash   -e 'input {  stdin{} } output { file { path => "/tmp/log-%{+YYYY.MM.dd}messages.txt"}}'
haha

[root@centos7-1 ~]# cd /tmp/
[root@centos7-1 tmp]# ls
log-2018.08.04messages.txt 
[root@centos7-1 tmp]# cat log-2018.08.04messages.txt 
{"host":"centos7-1","message":"haha","@version":"1","@timestamp":"2018-08-04T04:23:15.900Z"}

 3.1.3编写配置文件收集系统日志并写入到elasticsearch服务器

[root@centos7-1 ~]# cat /etc/logstash/conf.d/systemlog.conf 
input {
    file {
      path => "/var/log/messages"   #收集日志路径
      start_position => "beginning" #第一次从头收集，之后从新添加的日志收集
      type => "systemlog-cent7.1"   #名称
      stat_interval => "2"          #多久收集一次
    }    
}

output {
    elasticsearch {
      hosts => ["192.168.10.10:9200"]                     #主机地址
      index => "logstash-system-log-cent7-%{+YYYY.MM.dd}" #名称 
    }
    file {
      path => "/tmp/systemlog.txt"
    }
}

 验证文件

[root@centos7-1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/systemlog.conf -t
Configuration OK
[INFO ] 2018-08-04 12:44:11.922 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
增加权限

 

3.1.4 启动logstash

[root@centos7-1 ~]# systemctl enable logstash.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
[root@centos7-1 ~]# systemctl start logstash.service 
[root@centos7-1 ~]# systemctl status logstash.service 
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: active (running) since 六 2018-08-04 12:50:23 CST; 32s ago
 Main PID: 110252 (java)
    Tasks: 13
   CGroup: /system.slice/logstash.service
           └─110252 /bin/java -Xms1g -Xmx1g -XX:+UseParN...

8月 04 12:50:23 centos7-1 systemd[1]: Started logstash.
8月 04 12:50:23 centos7-1 systemd[1]: Starting logstash...
8月 04 12:50:23 centos7-1 logstash[110252]: OpenJDK 64-...
Hint: Some lines were ellipsized, use -l to show in full.
[root@centos7-1 ~]# tail -f /var/log/logstash/logstash-plain.log 
[2018-08-04T12:51:43,820][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}

3.1.5 验证本地文件

8-08-04T04:57:06.523Z","path":"/var/log/messages"}
[root@centos7-1 tmp]# pwd
/tmp
[root@centos7-1 tmp]# cat systemlog.txt 
{"@version":"1","host":"centos7-1","type":"systemlog-cent7.1","message":"Aug  4 12:40:02 centos7-1 systemd: Started Session 915 of user root.","@timestamp":"2018-08-04T04:57:02.151Z","path":"/var/log/messages"}
{"@version":"1","host":"centos7-1","type":"systemlog-cent7.1","message":"Aug  4 12:40:02 centos7-1 systemd: Starting Session 915 of user root.","@timestamp":"2018-08-04T04:57:02.230Z","path":"/var/log/messages"}
{"@version":"1","host":"centos7-1","type":"systemlog-cent7.1","message":"Aug  4 12:47:29 centos7-1 systemd: [/usr/lib/systemd/system/firstboot-graphical.service:14] Support for option SysVStartPriority= has been removed and it is ignored","@timestamp":"2018-08-04T04:57:02.231Z","path":"/var/log/messages"}

3.1.6 验证 elasticsearch