Logstash基础入门部署
Logstash环境准备与安装
环境准备
| 主机名 |
外网IP |
内网IP |
角色 |
应用 |
| ELKstack01 |
10.0.0.81 |
172.16.1.81 |
ES日志存储数据库 |
JDK、elasticsearch |
| ELKstack02 |
10.0.0.82 |
172.16.1.82 |
ES日志存储数据库 |
JDK、elasticsearch |
| ELKstack03 |
10.0.0.83 |
172.16.1.83 |
日志数据转发工具 |
JDK、logstash |
安装
# 1.安装jdk
[root@elkstack03 ~]# yum install -y java
# 2.安装Logstash
[root@elkstack03 ~]# yum localinstall -y logstash-5.6.16.rpm
# 3.Logstash工作目录授权
[root@elkstack03 ~]# chown -R logstash.logstash /usr/share/logstash/
Logstash的插件
|
|
|
| INPUT支持事件源 |
OUTPUT支持输出源 |
CODEC编解码器支持编码 |
| azure_event_hubs(微软云事件中心) |
elasticsearch(搜索引擎数据库) |
avro(数据序列化) |
| beats(filebeat日志收集工具) |
email(邮件) |
CEF(嵌入式框架) |
| elasticsearch(搜索引擎数据库) |
file(文件) |
es_bulk(ES中的bulk api) |
| file(文件) |
http(超文本传输协议) |
Json(数据序列化、格式化) |
| generator(生成器) |
kafka(基于java的消息队列) |
Json_lines(便于存储结构化) |
| heartbeat(高可用软件) |
rabbitmq(消息队列 OpenStack) |
line(行) |
| http_poller(http api) |
redis(缓存、消息队列、NoSQL) |
multiline(多行匹配) |
| jdbc(java连接数据库的驱动) |
s3*(存储) |
plain(纯文本，事件间无间隔) |
| kafka(基于java的消息队列) |
stdout(标准输出) |
rubydebug(ruby语法格式) |
| rabbitmq(消息队列 OpenStack) |
tcp(传输控制协议) |
|
| redis(缓存、消息队列、NoSQL) |
udp(用户数据报协议) |
|
| s3*(存储) |
|
|
| stdin(标准输入) |
|
|
| syslog(系统日志) |
|
|
| tcp(传输控制协议) |
|
|
| udp(用户数据报协议) |
|
|
Logstash输入输出测试
从标准输入到标准输出
[root@elkstack03 ~]# /usr/share/logstash/bin/logstash -e 'input{ stdin{} } output{ stdout{} }'
The stdin plugin is now waiting for input:
nginx log // 标准输入
{ // 标准输入
"@version" => "1",
"host" => "elkstack03",
"@timestamp" => 2022-09-06T14:37:40.541Z,
"message" => "nginx log"
}
xxx // 标准输入
{ // 标准输入
"@version" => "1",
"host" => "elkstack03",
"@timestamp" => 2022-09-06T14:38:11.133Z,
"message" => "xxx"
}
从标准输入到文件中
[root@elkstack03 ~]# /usr/share/logstash/bin/logstash -e 'input{ stdin{} } output{ file{ path => "/tmp/wc.log" } }'
从标准输入到ES
[root@elkstack03 ~]# /usr/share/logstash/bin/logstash -e 'input{ stdin{} } output{ elasticsearch{ hosts => ["10.0.0.81:9200"] index => "wc_access_log-2022-09-06" } }'
``
## Logstash收集系统日志
### 将文件中的日志放入ES
```bash
[root@elkstack03 ~]# /usr/share/logstash/bin/logstash -e 'input{ file{ type => "wc" path => "/var/log/messages" start_position => "beginning" }} output{ elasticsearch{ hosts => ["10.0.0.81:9200"] index => "message_log-2022-09-06" } }'
[root@elkstack03 ~]# cd /etc/logstash/conf.d/
[root@elkstack03 conf.d]# vim message_file_es.conf
input{
file{
type => "msg_log"
path => "/var/log/messages"
start_position => "beginning"
}
}
output{
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls_msg_log-2022-09-06"
}
}
## 启动Logstash指定配置文件
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf
## 检测配置文件语法
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf -t
Configuration OK
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf &
Logstash收集多个日志到ES
[root@elkstack03 conf.d]# systemctl stop logstash
[root@elkstack03 conf.d]# vim message_file_es.conf
input{
file{
type => "msg_log"
path => "/var/log/messages"
start_position => "beginning"
}
file{
type => "sec_log"
path => "/var/log/secure"
start_position => "beginning"
}
}
output{
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls_msg_log-2022-09-06"
}
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls_sec_log-2022-09-06"
}
}
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf &
## 优化日期
[root@elkstack03 conf.d]# vim message_file_es.conf
input{
file{
type => "msg_log"
path => "/var/log/messages"
start_position => "beginning"
}
file{
type => "sec_log"
path => "/var/log/secure"
start_position => "beginning"
}
}
output{
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls2_msg_log-%{yyyy.MM.dd}"
}
elasticsearch{
hosts => ["10.0.0.81:9200"]
index => "zls2_sec_log-%{yyyy.MM.dd}"
}
}
[root@elkstack03 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/message_file_es.conf &
## 优化index名
[root@elkstack03 conf.d]# vim message_file_es.conf
input{
file{
type => "msg"
path => "/var/log/messages"
start_position => "beginning"
}
file{
type => "sec"
path => "/var/log/secure"
start_position => "beginning"
}
}
output{
elasticsearch{：
hosts => ["10.0.0.81:9200"]
index => "%{type}-%{+yyyy.MM.dd}"
}
}
