Logstash基础入门部署
Logstash环境准备与安装
环境准备
主机名 |
外网IP |
内网IP |
角色 |
应用 |
ELKstack01 |
10.0.0.81 |
172.16.1.81 |
ES日志存储数据库 |
JDK、elasticsearch |
ELKstack02 |
10.0.0.82 |
172.16.1.82 |
ES日志存储数据库 |
JDK、elasticsearch |
ELKstack03 |
10.0.0.83 |
172.16.1.83 |
日志数据转发工具 |
JDK、logstash |
安装
# 1.安装jdk
[root@elkstack03 ~]# yum install -y java
# 2.安装Logstash
[root@elkstack03 ~]# yum localinstall -y logstash-5.6.16.rpm
# 3.Logstash工作目录授权
[root@elkstack03 ~]# chown -R logstash.logstash /usr/share/logstash/
Logstash的插件
|
|
|
INPUT支持事件源 |
OUTPUT支持输出源 |
CODEC编解码器支持编码 |
azure_event_hubs(微软云事件中心) |
elasticsearch(搜索引擎数据库) |
avro(数据序列化) |
beats(filebeat日志收集工具) |
email(邮件) |
CEF(嵌入式框架) |
elasticsearch(搜索引擎数据库) |
file(文件) |
es_bulk(ES中的bulk api) |
file(文件) |
http(超文本传输协议) |
Json(数据序列化、格式化) |
generator(生成器) |
kafka(基于java的消息队列) |
Json_lines(便于存储结构化) |
heartbeat(高可用软件) |
rabbitmq(消息队列 OpenStack) |
line(行) |
http_poller(http api) |
redis(缓存、消息队列、NoSQL) |
multiline(多行匹配) |
jdbc(java连接数据库的驱动) |
s3*(存储) |
plain(纯文本,事件间无间隔) |
kafka(基于java的消息队列) |
stdout(标准输出) |
rubydebug(ruby语法格式) |
rabbitmq(消息队列 OpenStack) |
tcp(传输控制协议) |
|
redis(缓存、消息队列、NoSQL) |
udp(用户数据报协议) |
|
s3*(存储) |
|
|
stdin(标准输入) |
|
|
syslog(系统日志) |
|
|
tcp(传输控制协议) |
|
|
udp(用户数据报协议) |
|
|
Logstash输入输出测试
从标准输入到标准输出
[root@elkstack03 ~]# /usr/share/logstash/bin/logstash -e 'input{ stdin{} } output{ stdout{} }'
The stdin plugin is now waiting for input:
nginx log // 标准输入
{ // 标准输入
"@version" => "1",
"host" => "elkstack03",
"@timestamp" => 2022-09-06T14:37:40.541Z,
"message" => "nginx log"
}
xxx // 标准输入
{ // 标准输入
"@version" => "1",
"host" => "elkstack03",
"@timestamp" => 2022-09-06T14:38:11.133Z,
"message" => "xxx"
}
![](https://img2022.cnblogs.com/blog/2774129/202209/2774129-20220906191849031-1431401049.png)
从标准输入到文件中
[root@elkstack03 ~]# /usr/share/logstash/bin/logstash -e 'input{ stdin{} } output{ file{ path => "/tmp/wc.log" } }'
![](https://img2022.cnblogs.com/blog/2774129/202209/2774129-20220906192016856-729533523.png)
从标准输入到ES
[root@elkstack03 ~]# /usr/share/logstash/bin/logstash -e 'input{ stdin{} } output{ elasticsearch{ hosts => ["10.0.0.81:9200"] index => "wc_access_log-2022-09-06" } }'
``
![](https://img2022.cnblogs.com/blog/2774129/202209/2774129-20220906192147911-1666505451.png)
## Logstash收集系统日志
### 将文件中的日志放入ES
```bash
[root@elkstack03 ~]# /usr/share/logstash/bin/logstash -e 'input{ file{ type => "wc" path => "/var/log/messages" start_position => "beginning" }} output{ elasticsearch{ hosts => ["10.0.0.81:9200"] index => "message_log-2022-09-06" } }'
![](https://img2022.cnblogs.com/blog/2774129/202209/2774129-20220906192552976-1613581695.png)
[root@elkstack03 ~]# cd /etc/logstash/conf.d/
[root@elkstack03 conf.d]# vim message_file_es.conf
input{
file{
type => "msg_log"
path => "/var/log/messages"
start_position => "beginning"
}
}
output{
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls_msg_log-2022-09-06"
}
}
## 启动Logstash指定配置文件
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf
## 检测配置文件语法
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf -t
Configuration OK
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf &
Logstash收集多个日志到ES
[root@elkstack03 conf.d]# systemctl stop logstash
[root@elkstack03 conf.d]# vim message_file_es.conf
input{
file{
type => "msg_log"
path => "/var/log/messages"
start_position => "beginning"
}
file{
type => "sec_log"
path => "/var/log/secure"
start_position => "beginning"
}
}
output{
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls_msg_log-2022-09-06"
}
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls_sec_log-2022-09-06"
}
}
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf &
## 优化日期
[root@elkstack03 conf.d]# vim message_file_es.conf
input{
file{
type => "msg_log"
path => "/var/log/messages"
start_position => "beginning"
}
file{
type => "sec_log"
path => "/var/log/secure"
start_position => "beginning"
}
}
output{
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls2_msg_log-%{yyyy.MM.dd}"
}
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls2_sec_log-%{yyyy.MM.dd}"
}
}
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf &
## 优化index名
[root@elkstack03 conf.d]# vim message_file_es.conf
input{
file{
type => "msg"
path => "/var/log/messages"
start_position => "beginning"
}
file{
type => "sec"
path => "/var/log/secure"
start_position => "beginning"
}
}
output{
elasticsearch{:
hosts => ["10.0.0.81:9200"]
index => "%{type}-%{+yyyy.MM.dd}"
}
}