3.logstash

1.什么是logstash

2.为什么用logstash

3.logstash架构介绍

4.安装logstash

5.logstash配置

• 5.1 input插件

  • 5.1.1 stdin模块

  • 5.1.2 file模块

  • 5.1.3 beat模块

  • 5.1.4 http模块

  • 5.1.1 kafka模块

• 5.2 filter插件

1.什么是logstash

logstash是开源的数据处理管道 它集多个源数据收集 数据处理 数据输出的工具 由java编写

2.为什么用logstash

采集的数据不是json格式的，kibana无法分字段展示，无法有效的观察数据，所以需要把数据转化成json格式！把“非结构化数据”转化成“结构化数据”！

3.logstash架构介绍

logstash 基础架构 类似于pipeline流水线，如下图所示

• input： 数据采集 （常用的插件 beat http stdin file kafka ）
• filter： 数据处理 (常用的插件 grok geoip date useragent mutate )
• output： 数据输出 (elasticsearch rubydebug )

4.安装logstash

4.1 下载安装包

[root@rongbiz-43 ~]# wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.17.8/logstash-7.17.8-x86_64.rpm

4.2 安装rpm包

[root@rongbiz-43 ~]# rpm -ivh elasticsearch-7.8.1-x86_64.rpm 
warning: elasticsearch-7.8.1-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
	package elasticsearch-0:7.8.1-1.x86_64 is already installed

5.logstash配置

5.1 input插件

5.1.1 stdin 模块

[root@rongbiz-43 conf.d]# vim stdin_logstash.conf
input {
    stdin {
      type => "stdin"          #自定义事件类型,可用于后续判断。
      tags => "stdin_type"     #自定义事件tag,可用于后续判断
    }
}
output {
    stdout {
       codec => "rubydebug"
   }
}

运行

[root@rongbiz-43 conf.d]# /usr/share/logstash/bin/logstash  -f  /etc/logstash/conf.d/stdin_logstash.conf  -r
yangtao       #输入yangtao的字符
{             #回显的信息
      "@version" => "1",
          "tags" => [
        [0] "stdin_type"
    ],
          "host" => "rongbiz-43",
          "type" => "stdin",
    "@timestamp" => 2023-01-06T02:48:57.892Z,
       "message" => "yangtao"
} 

5.1.2 file 模块

[root@rongbiz-43 conf.d]# vim file_logstash.conf

input {
   file {
     type => syslog
     path => "/var/log/messages"
     exclude => "*.gz"               #不监听的文件排除
     start_position => "beginning"   #从文件头部读取 beginning  或者end
     stat_interval  => "3"           #检查同步周期，默认1s
  }
}
output {
   stdout {
     codec => rubydebug
   }
}

运行

[root@rongbiz-43 conf.d]# /usr/share/logstash/bin/logstash  -f  /etc/logstash/conf.d/file_logstash.conf 
[INFO ] 2023-01-06 11:15:20.329 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2023-01-06 11:15:20.433 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2023-01-06 11:15:20.430 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[WARN ] 2023-01-06 11:15:20.747 [[main]<file] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
{
    "@timestamp" => 2023-01-06T03:15:20.824Z,
          "path" => "/var/log/messages",
          "host" => "rongbiz-43",
      "@version" => "1",
       "message" => "Jan  6 11:14:44 rongbiz-43 systemd: Stopping nginx - high performance web server...",
          "type" => "syslog"
}
{
    "@timestamp" => 2023-01-06T03:15:20.780Z,
          "path" => "/var/log/messages",
          "host" => "rongbiz-43",
      "@version" => "1",
       "message" => "1",
          "type" => "syslog"
}

5.1.3 beat 模块

[root@rongbiz-43 conf.d]# vim beat_logstash.conf

input {
   beats {
      port => 5044
   }
}
output {
   stdout {
     codec => rubydebug
   }
}

5.1.4 http 模块

5.1.5 kafka 模块

5.2 filter插件

·111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111112111111111111211112111111111111111111111111111111111111111111111111111111111111111